JP7235689B2 - Haptic sensation presentation method, system and program - Google Patents

Description

The present invention relates to a tactile sensation presentation method, system, and program, and in particular, a tactile sensation presentation method that can visually and intuitively present to the user the tactile sensation of any object displayed at a position on the display specified by the user. Regarding systems and programs.

Non-Patent Document 1 discloses a system that displays a position input by a pointing device such as a mouse as an arrow cursor on a display, and presents a tactile sensation to the user by controlling the movement and size of the cursor. For example, if you try to move the arrow cursor over a rickety surface displayed on the display, the mouse cursor will appear to wobble up and down. Also, if you try to move the arrow cursor at the part where the adhesive surface of the packing tape is displayed, the cursor will move while being stuck. In Non-Patent Document 1, the user can obtain a simulated tactile sensation of an object displayed on the display by such a change in cursor movement.

Japanese Patent Laid-Open No. 2002-200011 describes a server device that generates a web page providing various information about products in the form of an HTML document, mainly for use in online shopping, and a device that drives an actuator based on touch input from a touch panel. Disclosed is a tactile sensation presentation device that presents the tactile sensation of an object.

In Patent Document 1, a web page generated by a server device displays an icon labeled "tactile sensation", and a user clicks the icon from a consumer terminal such as a smartphone to present a tactile sensation from the server device. Device control parameters can be downloaded. The consumer terminal and the tactile sensation presentation device are connected by wire or wirelessly, the downloaded control parameters are applied to the tactile sensation presentation device, and the user can experience the tactile sensation of the desired product by touching the touch panel of the tactile sensation presentation device with a fingertip.

Japanese Patent Application Laid-Open No. 2019-128738

Keita Watanabe Michiaki Yasumura, VisualHaptics: Generating Haptic Sensation Using Only Visual Cues,ACE '08: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology

The device of Patent Literature 1 is optimized for use in providing tactile sensations to users in online shopping. Non-Patent Document 1 is also a system that presents the tactile sensation of an object prepared in advance. Therefore, when constructing a device that visually and intuitively provides a user with a tactile sensation of an arbitrary object in an arbitrary scene, the following points become technical problems.

First, since it is necessary to prepare tactile parameters linked to objects in advance, as the number of target objects for which tactile sensations are experienced increases, the number of times the tactile parameters are set increases, resulting in a time cost.

Secondly, since one tactile parameter is set for one article, the tactile sensations of a plurality of objects cannot be reproduced simultaneously in the same scene. For example, in a kitchen scene, there are various objects such as cups and frying pans. In such a scene, it is impossible to visually and intuitively express the tactile sensation of a cup and the tactile sensation of a frying pan at the same time.

An object of the present invention is to solve the above technical problems, and to provide a tactile sensation presentation method, system, and program that can visually and intuitively present the tactile sensation of any object displayed on a display requested by the user. to do.

In order to achieve the above objects, the present invention is characterized by having the following configuration in a tactile sensation presentation system that visually presents the tactile sensation of an object displayed on a display.

(1) means for acquiring an image of an object, tactile sensation storage means for holding tactile sensation parameters for each type of object, means for identifying the type and position of the object displayed on the display, and identifying the type and position of the object. means for generating tactile information that associates each position on the display with a tactile parameter based on the identification result and the tactile sensation storage means; means for detecting a tactile sensation requesting position based on a tactile sensation requesting operation on the display; and the tactile sensation A means for acquiring a tactile parameter corresponding to the requested position based on the tactile information, and a means for displaying a tactile indicator visually expressing the tactile sensation at the tactile requested position based on the acquired tactile parameter.

(2) Multiple tactile indicators are displayed over the entire background area of the object.

(3) When the tactile sensation request position slides, the tactile indicator changes according to the tactile sensation at each slide position.

(4) The display position of the tactile indicator is moved according to the sliding of the tactile request position.

(5) A means is provided for correcting the tactile information generated based on the current frame image based on each tactile information generated based on each frame image before the last time.

(6) Means for correcting all pixels in the current frame or types of tactile sensation requested positions by a weighted linear sum based on the distance between each pixel or tactile sensation requested position and each pixel of each frame image before the previous one and the type thereof. Equipped with

(7) A type histogram is created for each frame, and the correction means is not applied between frames in which the change in the type histogram exceeds a predetermined threshold.

(1) The type and position of the object displayed on the display are identified, and when the user designates the display position of the object that the user wants to experience the tactile sensation, the tactile parameter is determined based on the identification result of the object displayed at the designated position. Since the obtained tactile sensation is represented by the indicator, the user can visually and intuitively obtain the tactile sensation of any object displayed on the display.

As a result, it becomes possible to realize movies with tactile sensations, online shopping, etc., in which a product can be selected while visually experiencing the tactile sensation of each product included in the image. Moreover, since processing can be performed in real time, a telecommunication system with tactile sensation can be realized. Furthermore, by identifying fictitious characters and giving them tactile sensations, it is possible to produce applications with a more realistic feel, such as production of comic books, animations, etc., in which characters appearing can be touched.

(2) Since the tactile indicator is displayed on the entire background area of the object, various tactile sensations of the object can be presented visually and intuitively without obstructing the display of the object.

(3) When the tactile sensation requesting position slides, the tactile indicator changes according to the tactile sensation at each slide position, so that the tactile sensation of the article, which varies depending on the position, can be visually expressed without omission.

(4) Since the display position of the tactile indicator moves according to the sliding of the tactile sensation request position, the texture of the article can be dynamically expressed visually.

(5) The tactile information generated based on the current frame image can be corrected based on the tactile information generated based on the previous and previous frame images. It becomes possible to present the correct tactile sensation of the object to the user.

(6) Each pixel of the current frame or the type of the tactile sensation requested position is corrected by a weighted linear sum based on the distance between each pixel or the tactile sensation requested position and each pixel of each frame image before the previous time and the type thereof, The type estimation result can be abstracted temporally and spatially. Therefore, even if temporary or partial misrecognition occurs in estimating the type, the influence on the tactile expression can be reduced.

(7) A type histogram is created for each frame, and the correction is not applied between frames where the change in the type histogram exceeds a predetermined threshold. It is possible to prevent erroneous correction, such as correction based on later tactile information.

1 is a functional block diagram of a tactile presentation system to which the present invention is applied; FIG. FIG. 4 is a diagram showing an example of tactile information; It is a figure explaining the request|requirement method of a tactile sense. FIG. 4 is a diagram showing a method of displaying a tactile indicator according to the first embodiment of the present invention; FIG. 10 is a diagram showing a method of displaying a tactile indicator according to the second embodiment of the present invention; FIG. 10 is a diagram showing an example of expressing a tactile sensation with the shape of a tactile indicator; FIG. 10 is a diagram showing an example of expressing the tactile sense of a hedgehog in the shape of a tactile indicator. FIG. 10 is a diagram showing an example of expressing a tactile sensation with the size of a tactile indicator; FIG. 4 is a diagram expressing the tactile sensation of the cheek portion of a human face by the size of the tactile indicator. FIG. 10 is a diagram showing an example of expressing temperature by the color of a tactile indicator; FIG. 10 is a diagram showing an example of expressing smoothness by movement of a tactile indicator; FIG. 10 is a diagram showing an example of expressing a tactile sensation of a slide position of a touch operation by movement of an indicator; FIG. 11 is a diagram showing an example of expressing a tactile sensation when a touch operation is linearly slid by displacing an indicator to the left and right in the sliding direction; FIG. 10 is a diagram showing an example in which the tactile indicator in the background area is moved in synchronization with the sliding of the touch operation, and the shape and the like are changed according to the tactile sensation at the slide position. FIG. 10 is a diagram showing an example of partially switching the display of the tactile indicator in the process of changing the tactile sensation when the position of the touch operation is slid; FIG. 10 is a diagram schematically showing a display switching method of a tactile indicator when a touch position is slid between articles having different tactile sensations; 4 is a flow chart of a tactile presentation method to which the present invention is applied; 1 is a functional block diagram of an embodiment in which a tactile sensation presentation system is configured with a local terminal and a server device; FIG. 19 is a sequence flow of the tactile sensation presentation system shown in FIG. 18; 1 is a functional block diagram of an embodiment in which a tactile sensation presentation system is configured with a local terminal and a server device; FIG. 21 is a sequence flow of the tactile sensation presentation system shown in FIG. 20; 1 is a functional block diagram of an embodiment in which a tactile sensation presentation system is configured by a local terminal; FIG. FIG. 4 is a functional block diagram of a tactile sensation presentation system having a map correction unit; It is a figure explaining the function of a map correction part. FIG. 3 is a functional block diagram of a tactile presentation system in which a local terminal is provided with a map correction unit; FIG. 11 is a diagram schematically showing a method of correcting the type of tactile sensation requested position of the current frame; FIG. 4 is a diagram schematically showing a method of detecting switching of cameras and replacement of objects; FIG. 3 is a functional block diagram of a tactile sensation presentation system in which a map correction unit is provided in a server device;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of a tactile presentation system 10 to which the present invention is applied. The tactile sensation presentation system 10 of the present invention presents tactile sensations to the user by visually expressing tactile information of an object specified by the user on a display that displays a plurality of objects. It should be noted that the following embodiment is an example that embodies the present invention, and does not limit the technical scope of the present invention.

In the present invention, an example will be described in which an article requiring a tactile sensation is designated by a touch operation on a touch panel display, but the present invention is not limited to this, and is designated by a pointing operation using a pointing device such as a mouse. You can do it.

The image acquisition unit 100 acquires a still image or a moving image including an object as a subject (hereinafter, represented by an image or a frame image) via a network or locally, and a display having a touch panel 102a capable of detecting touch operations on its display surface. 102.

The object identification unit 101 includes a type identification unit 101a and a position identification unit 101b, and identifies the type and position coordinates of the object displayed on the touch panel display 102, respectively. The positional coordinates may be planar 2D positional coordinates, or may be stereoscopic 3D positional coordinates to which depth information (z-axis direction) is added.

As the type identifying unit 101a, an algorithm such as YOLO for estimating two-dimensional positional coordinates as a rectangular area or DeepLab for estimating two-dimensional positional coordinates as a free-form area can be used. PlaneNet or the like can be used as an algorithm for estimating depth information. Also, if the depth information of the object is acquired by a TOF (Time of Flight) sensor or the like, that value may be used.

A tactile database (DB) 103 holds tactile parameters, which will be described in detail later, for each type of object. The tactile parameters include parameters corresponding to material properties such as "Young's modulus", "surface roughness" or "thermal conductivity" unique to each object, and/or "hardness", "roughness" or " It includes a parameter corresponding to the perceived strength of the tactile sensation, such as "cool feeling".

The tactile information generation unit 104 acquires from the tactile sense database 103 tactile sense parameters corresponding to the type of each object acquired from the object identification unit 101, and generates a recognized object map d1 and a tactile sense data subset as shown in FIG. d2 and tactile information D is generated. The recognized object map d1 defines the position coordinates of each object on the touch panel display 102. FIG. The tactile data subset d2 associates each object type with tactile parameters.

The tactile sensation request detection unit 105 includes a position detection unit 105a and a touch method detection unit 105b. The position detection unit 105a detects the position of the object for which the user is requesting the tactile sensation as the tactile sensation request position P based on the user's touch operation on the touch panel 102a of the touch panel display 102 displaying the object. The touch detection unit 105b detects a time-series change in touch speed or touch pressure as a "touch Q" based on the touch operation. If the tactile sensation request is made by a pointing device such as a mouse instead of a touch operation, only the tactile sensation request position P is detected.

The tactile parameter acquisition unit 106 acquires the tactile parameter of the object for which the user has requested the tactile sensation from the tactile information D based on the tactile sensation request position P. The tactile indicator generation unit 107 generates a tactile indicator by applying the tactile sensation request position P, the touch method Q, and the tactile sensation parameter to a predetermined model formula learned in advance. A tactile indicator display unit 108 superimposes a tactile indicator on the frame image.

FIGS. 3, 4, and 5 are diagrams showing a display method of the tactile indicator In according to the present invention. As shown in FIG. 3, on a screen on which a car, a person, and a dog are displayed, the operator touches the cheek of a human face. When a tactile sensation is requested by performing a touch operation or the like on a part, the tactile parameter of the tactile sensation request position P is acquired by the tactile parameter acquisition unit 106 .

FIG. 4 is a diagram showing a display method of the tactile indicator In according to the first embodiment of the present invention. In the present embodiment, a tactile sensation indicator In having a predetermined shape that visually expresses the tactile sensation of the tactile sensation requesting position P is superimposed on the tactile sensation requesting position P in the center.

FIG. 5 is a diagram showing a display method of a tactile indicator according to the second embodiment of the invention. In this embodiment, a large number of tactile sense indicators In that visually represent the tactile sense of the tactile sense requesting position P are discretely and regularly arranged in the background region (including the car region and the dog region) other than the article at the tactile sense requesting position P. It is displayed superimposed in an array. At this time, in order to make the tactile indicator In easier to see, the display color of an article that does not require a tactile sensation may be made lighter or lighter.

The indicator In can visually express the tactile sensation of the tactile sensation requesting position P by its shape, color, size, movement, and the like. FIG. 6 is a diagram showing an example of expressing a tactile sensation with the shape of the tactile indicator In. If the tactile sensation is sufficiently smooth, the tactile indicator In has a circular shape that intuitively expresses the smoothness. As the surface becomes rougher, protrusions appear on the circumference, and the shape of the protrusions becomes sharper to intuitively express the roughness of the surface.

FIG. 7 is a diagram showing a visualization example of the tactile sensation when a tactile sensation is requested to the body surface of a hedgehog in the second embodiment, and the tactile indicator In displayed in the background area is sharpened to the outer periphery. A large number of protrusions appear.

FIG. 8 shows an example of expressing the hardness of the tactile sensation required position P by the size of the tactile indicator In. In the case where the touch panel sensor 102a is configured to detect contact pressure, in the present embodiment, the size of the tactile indicator In increases as the pressing pressure during the touch operation increases and as the tactile sensation becomes softer. FIG. 9 is a diagram expressing the hardness of the cheek portion of the human face by the size of the tactile indicator In. Compared to FIG. 5, the tactile indicator In is displayed larger to express that the tactile sensation is softer.

FIG. 10 is a diagram showing an example in which the temperature of the tactile sensation requesting position P is represented by the tactile indicator In. When the tactile information includes temperature information, the difference in temperature is visualized by the color of the tactile indicator In. In this embodiment, the coldest tactile sensation is expressed in purple, and as the temperature increases, the color changes to indigo, blue, green, yellow, orange, and red.

FIG. 11 is a diagram showing an example of expressing the smoothness of the required tactile position P by the movement of the tactile indicator In, and the smoothness of the required tactile position P is expressed by the amplitude and/or frequency of the tactile indicator In. . In this embodiment, the rougher the tactile sensation at the tactile sensation requesting position P, the larger the amplitude/lower the vibration frequency of the tactile indicator In. Note that the amplitude and/or frequency of the tactile indicator In may change according to the distance from the tactile sensation requesting position P. For example, the farther the tactile indicator In is from the tactile sensation requesting position P, the more the amplitude and/or frequency of the tactile indicator In can be suppressed. Also, by giving a time delay to the vibration according to the distance to the tactile sensation request position P, elasticity and softness like rubber can be expressed.

In the above embodiment, the tactile indicator In is represented by any one of shape, color, size, and movement depending on the tactile sensation, but the present invention is not limited to this. For example, the smoothness, temperature and softness of the tactile sensation requested position P may be expressed by a combination of the shape, color and size of the tactile indicator In.

Furthermore, in the above-described embodiment, the method of visually expressing a tactile sensation has been described by taking as an example the case where a touch operation or the like is performed at a fixed location on the article. When slid, the shape or the like of the indicator In may be changed continuously according to the tactile sensation of each slide position.

FIG. 12 is a diagram showing an example of expressing a tactile sensation by not synchronizing the movement of the indicator In with the tactile sensation of the sliding position when the position of the touch operation or the like is slid. In this embodiment, if the tactile sensation of the slide position is sufficiently smooth, the slide motion of the touch position and the movement of the indicator In are synchronized as shown in FIG. On the other hand, if the tactile sensation at the slide position is not smooth, as shown in FIG. .

FIG. 13 is a diagram showing an example in which when the position of a touch operation or the like is linearly slid, the indicator In moves while being displaced left and right in the slide direction according to the tactile sensation of the slide position. In the present embodiment, the rougher the tactile sensation of the slide position, the greater the amount of lateral displacement.

FIG. 14 shows the shape of the tactile indicator In that, when the position of the touch operation or the like is slid, moves the tactile indicator In in the background area in the same direction or in the opposite direction to the sliding direction in synchronization with the movement, and further moves the tactile indicator In. 4 is a diagram showing an example in which , size, color, etc. are changed according to the tactile sensation of the slide position. FIG.

FIG. 15 is a diagram showing an example of partially switching the display of the tactile indicator In in the process of changing the tactile sensation when the position of the touch operation or the like is slid. A case of performing a slide operation to the surface will be described as an example.

First, when a touch operation on a person's hair is detected, a circular tactile indicator In that visualizes a smooth tactile sensation is displayed on the entire background area, as shown in FIG. Next, when the touch position slides in the front right direction and reaches a position straddling the hair and the body surface of the dog as shown in FIG. While a tactile indicator In that visualizes is displayed, a tactile indicator In that visualizes "roughness" is displayed in the background on the right side of the screen corresponding to the dog's body surface.

When the touch position moves further to the right and reaches the body surface of the dog, all the tactile indicators In displayed in the background area other than the dog change their shape and color to that of the dog, as shown in FIG. It changes its shape and color according to the tactile sensation of the body surface.

FIG. 16 is a diagram schematically showing a display switching method of the tactile indicator In when the position of a touch operation or the like is slid between articles having different tactile sensations.

In this embodiment, the tactile information D in the vicinity of the tactile sensation request position is divided into small rectangular areas corresponding to the respective tactile indicator In, and the shape and color of each tactile indicator In are determined according to the tactile sensation of each rectangular area. . In this embodiment, the tactile sensation of each rectangular area in the left three columns is "hair", and the tactile sensation of each rectangular area in the right three columns is "dog". is displayed.

On the other hand, "hair" and "dog" are mixed in the central row, but since the area of "dog" is larger than the area of "hair", the tactile indicator In corresponding to "dog" is displayed. be done.

FIG. 17 is a flow chart showing the operation of the tactile sensation presentation system. In step S1, the image acquisition unit 100 acquires a frame-by-frame image of an object as a subject via a network or from an internal memory or the like.

In step S2, the image is displayed on the display 102. In parallel with or before or after this, the type identification unit 101a identifies the type of each object extracted from the frame image by the identification unit 101 in step S3, and in step S4 , the position identification unit 101b identifies the position coordinates of each object on the display 102, and in step S5, the tactile information generation unit 104 generates the tactile information D based on the identification result of the type and position of the object and the tactile database 103. Generate.

In step S<b>6 , the tactile sensation request detection unit 105 determines whether or not the user has touched the display 102 . When the touch operation is detected, the tactile request detection unit 105 recognizes the position as the tactile request position P, further detects the touch manner Q, and proceeds to step S7. In step S7, the tactile information D is referenced, and tactile parameters associated with the tactile sensation request position P on the recognition object map d1 are acquired from the tactile sensation data subset d2.

In step S8, the tactile sensation request position P, the touch manner Q, and the tactile sensation parameter are applied to the model formula to generate the tactile sensation indicator In. In step S9, the tactile indicator In is superimposed on the frame image and presented to the user. Therefore, the user can visually and intuitively feel the tactile sensation of any object displayed on the display.

According to this embodiment, the user can visually experience the tactile sensation of any object displayed on the touch panel display. It will be possible to realize selectable online shopping. Moreover, since processing can be performed in real time, a telecommunication system with tactile sensation can be realized. Furthermore, by identifying fictitious characters and giving them tactile sensations, it is possible to produce applications with a more realistic feel, such as production of comic books, animations, etc., in which characters appearing can be touched.

FIG. 18 is a functional block diagram showing a first distributed example of each function constituting the tactile sensation presentation system. Since the same reference numerals as above denote the same or equivalent parts, description thereof will be omitted. In the present embodiment, each function of the tactile sensation presentation system 10 is distributed to the local terminal 1 represented by a smartphone and the server device 2 on the network, and the tactile sensation presentation is realized by the cooperation of each function. is characterized by

In the local terminal 1, the tactile information acquisition unit 104a transmits the image information acquired by the image acquisition unit 100 to the server device 2, and transmits the tactile information D composed of the object identification map d1 and the tactile data set d2 to the server. Acquired from device 2 .

FIG. 19 is a sequence flow showing the operation of the first distribution example. At time t1, the video acquisition unit 100 of the local terminal 1 acquires a frame image of an object. At time t2, the tactile information acquisition unit 104a transmits a tactile information request including frame image information to the server device 2 via the network. The frame image is displayed on the touch panel display 102 at time t3.

When the server device 2 receives the tactile sensation information request, at time t4, the identification unit 101 extracts objects from the frame image, and further identifies the type and position coordinates of each object. At time t5, a parameter request describing the type of the object is transmitted from server device 2 to tactile DB 103 . The tactile DB 103 responds to the server device 2 with tactile parameters corresponding to the object type at time t6.

At time t7, the server device 2 generates the tactile information D described with reference to FIG. 2 based on the acquired tactile parameters and position coordinates of each object. to respond to.

In parallel with this, at time t9, the tactile sensation request detection unit 105 of the local terminal 1 determines the position of the object for which the user is requesting the tactile sensation as the tactile sensation request position P based on the user's touch operation on the touch panel unit 102b. , and the time-series changes in touch speed and touch pressure are detected as touch method Q.

In the local terminal 1, the tactile parameter acquisition unit 106 acquires the tactile parameter of the object requested by the user from the tactile information D based on the tactile request position P at time t10. At time t11, the tactile indicator generation unit 107 generates a tactile indicator In by applying the tactile sensation request position P, the touch manner Q, and the tactile sensation parameter to the model formula. At time t12, the tactile indicator display unit 108 superimposes the tactile indicator In on the current frame image.

According to this embodiment, since a plurality of functions required for tactile sensation presentation can be distributed to the local terminal 1 and the server device 2, a visual tactile sensation presentation function can be provided even to a local terminal with low processing capability.

FIG. 20 is a functional block diagram showing a second distributed example of each function constituting the tactile sensation presentation system. Since the same reference numerals as above denote the same or equivalent parts, description thereof will be omitted. This embodiment is characterized in that the identification unit 101 is implemented in the local terminal 1 as compared with the first distributed example.

FIG. 21 is a sequence flow showing the operation of the second distribution example. At time t21, the video acquisition unit 100 of the local terminal 1 acquires a frame image, and displays it on the touch panel display 102 at time t22. At time t23, the identification unit 101 extracts objects from the frame image, and identifies the type of each object and the position coordinates on the display 102. FIG.

At time t24, a tactile information request describing the type of the object is transmitted from the local terminal 1 to the server device 2. FIG. At time t25, a parameter request including the object type described in the tactile information request is transmitted from the server device 2 to the tactile DB 103. FIG. At time t26, the tactile sensation DB 3 responds to the server device 2 with the tactile sensation parameter corresponding to the object type in response to the parameter request. At time t27, the tactile information generation unit 104 of the server device 2 generates the tactile information D composed of the object identification map d1 and the tactile data set d2 based on the acquired tactile parameters and the positional coordinates of the object.

In parallel with this, at time t29, the tactile sensation request detection unit 105 of the local terminal 1 detects the tactile sensation request position P and the touch method Q based on the user's touch operation on the touch panel unit 102b. At time t30, the tactile parameter acquisition unit 106 acquires tactile parameters from the tactile information D based on the tactile sensation request position P. FIG.

At time t31, the tactile indicator generation unit 107 generates a tactile presentation signal by applying the tactile sensation request position P, the touch manner Q, and the tactile sensation parameter to the model formula. At time t32, the tactile indicator In is superimposed on the frame image.

According to this embodiment, although the processing load on the local terminal 1 related to tactile sensation presentation increases compared to the first distribution example, the amount of traffic between the local terminal 1 and the server device 2 can be reduced. .

FIG. 22 is a functional block diagram showing a third distribution example of each function constituting the tactile sensation presentation system. Since the same reference numerals as above denote the same or equivalent parts, the description thereof will be omitted. This embodiment is characterized in that all the functions of the tactile sensation presentation system 10 are implemented in the local terminal 1 .

FIG. 23 is a functional block diagram showing another configuration of the tactile sensation presentation system 10, which is characterized by having a map corrector 109 that corrects the recognition object map d1 generated by the tactile information generator 104. FIG.

The tactile information generation unit 104 generates the recognition object map d1 for each frame based on the identification result of the identification unit 101 if the image is a moving image. At this time, as shown in FIG. 24, the type of object is temporarily and partially erroneously recognized between frames due to the discrimination accuracy of the discrimination unit 101a, light and shade in the image, image noise, etc., and the recognized object map d1 part of the error may occur. Such erroneous recognition occurs remarkably particularly in an identification algorithm that estimates the type of an object based on a free form.

If the erroneously recognized region is designated as the tactile sensation request position P, an erroneous tactile sensation is presented based on the erroneous tactile parameters, and if the correct tactile sensation is presented based on the correct tactile parameters in the next frame, the A change in tactile sensation at the time becomes a factor of noise, and the quality of the presented tactile sensation is significantly degraded.

The map correction unit 109 replaces the recognition object map d1 of the current frame with the recognition object map d1 of the past in order to prevent the quality of the presentation tactile sensation from being degraded due to an error in part of the recognition object map d1 due to erroneous recognition of the object type. Correction is made based on the recognition object map d1 for multiple frames. As a result, even if the type of the object cannot be correctly identified, the correct tactile sensation of the object can be presented to the user.

FIG. 25 is a functional block diagram showing an example of distribution of each function in the tactile sensation presentation system having the function of the map correction unit 109. The local terminal 102 is characterized by the map correction unit 109a.

In addition to the recognition object map d1 of the current frame, the map correction unit 109a holds the recognition object map d1 of the past n frames (the past 4 frames in this embodiment), and as shown in FIG. When the required position P is detected, the identification result of, for example, 5 pixels square, 125 pixels in total, centered on the coordinates of the tactile sensation required position P, is extracted from each recognition object map d1. Then, starting from the coordinates of the tactile sensation request position P, a weighted linear sum of tag IDs based on time or inter-pixel distance is calculated for each pixel in the past frame. Here, the tag ID is an identifier unique to the identification result of each object.

In this embodiment, the position coordinates of the tactile sensation request position P in the recognition object map d1 of the current frame are (xt, yt), and the position coordinates (x, y) of each pixel in the recognition object map d1 of f frames before Let vxyf be the tag ID, go back k frames, extract tag IDs from a rectangular area of (2N+1) pixels×(2M+1) pixels, and calculate the linear sum. A linear sum E (T) for tag ID (T) can be expressed by the following equation (1). However, wxyf is the linear sum weight from the tactile sensation request position P (xt, yt) in the current frame to each pixel coordinate (x, y) in the recognition object map j1 f frames ago, and f = 0 is the current frame. show.

Then, among the linear sums obtained for each tag ID, the linear sum with the largest value and the tag ID corresponding to it are selected. xt, yt) as the region of the selected tag ID. On the other hand, if the linear sum is smaller than the threshold, the position coordinates (xt, yt) are determined as the background area.

According to the present embodiment, since the estimation result is temporally and spatially abstracted, even if a temporary or partial recognition error occurs in the tactile information generation unit 104, the visual tactile presentation is not affected. can be reduced.

However, if the camera viewpoint is switched or the object in the video is changed, the correct estimation result may not be obtained if the map correction unit 109 calculates the weighted linear sum by going back to the past frames. There is

Here, as shown in FIG. 27, when a histogram is created using the identification result (tag ID) of the object in the recognition object map j1 as pixel data, this histogram shows which object occupies what area in the frame image. indicate whether Therefore, when the frame image showing "person", "dog", and "car" is switched to the frame image showing only "person", the histogram changes greatly as shown in the figure.

Therefore, in this embodiment, the value of the histogram for the tag ID is stored for the past several frames, and the histogram of the current frame is compared with the histogram of each past frame image to detect camera switching and object replacement. do.

For example, the rate of change between the past frame and the current frame is calculated for each tag ID, and the average value of the rate of change between tag IDs is calculated. If the calculated average value exceeds the threshold, it can be determined that camera switching or object replacement has occurred.

However, if the target area of the histogram is the entire frame, it is not possible to deal with partial variations of the frame. Therefore, in practice, it is desirable to divide one frame image into partial sections and calculate a histogram for each section to detect camera switching and object replacement in each section.

With the above method, when camera switching or object switching is detected in a certain area, if the user touches a part of that area, the weight of the linear sum is set again for the frame before detection, The type of the object at the touch position is determined again. This makes it possible to present visual tactile sensations that reduce the effects of erroneous recognition of objects, even in images in which cameras are switched or objects are replaced.

According to the present embodiment, it is possible to prevent erroneous correction, such as correcting tactile information before a large change in video based on tactile information after a large change. Further, according to this embodiment, only the tag ID of the pixel corresponding to the tactile sensation request position P is corrected based on the tactile information map M of the past frame, so the amount of calculation can be reduced. Therefore, implementation in the local terminal 1, which generally has low computing power, can be realized.

FIG. 28 is a functional block diagram showing another distribution example of each function in the tactile sensation presentation system having the function of the map correction unit 109. Map correction for realizing the function of the map correction unit 109 in the server device 2 is shown in FIG. It is characterized in that the part 109b is mounted.

In this embodiment, when the tactile sensation request position P is detected, only the tag ID of the pixel corresponding to the tactile sensation request position P is corrected based on the tactile sensation information map M of the past frame. P is unknown. Therefore, the map correction unit 109b corrects the recognition object map d1 by repeating the same procedure in advance for all pixels of the recognition object map generated for the current frame.

According to the present embodiment, the processing load on the local terminal 1 can be reduced because the correction processing of the recognition object map d1, which generally has a high processing load, can be performed by the server device.

1 Local terminal 2 Server device 10 Tactile presentation system 100 Video acquisition unit 101 Object identification unit 101 a Type identification unit 101 b Position identification unit 102 Touch panel display 103 Tactile database ( DB), 104... Tactile information generation unit, 105... Tactile request detection unit, 106... Tactile parameter acquisition unit, 107... Tactile indicator generation unit, 108... Tactile indicator display unit, 109, 109a, 109b... Map correction unit

Claims (15)

In a tactile sensation presentation system that visually presents the tactile sensation of an object displayed on a display,
means for obtaining an image of an object;
tactile sensation storage means for holding tactile sensation parameters for each type of object;
means for identifying the type and position of an object displayed on the display;
means for generating tactile information that associates each position on the display with a tactile parameter based on the result of identification of the type and position of the object and the tactile storage means;
means for detecting a tactile sensation requesting position based on a tactile sensation requesting operation on a display;
means for acquiring a tactile parameter corresponding to the tactile sensation request position based on the tactile sensation information;
means for displaying a tactile indicator that visually expresses the tactile sensation at the tactile sensation requested position based on the acquired tactile sensation parameter;
A tactile sensation presentation system, wherein a plurality of the tactile sensation indicators are displayed over the entire background area of the object . 2. The tactile sense presentation system according to claim 1 , wherein when the tactile sense request position slides, the tactile sense indicator changes according to the tactile sense of each slide position. 3. The tactile sense presentation system according to claim 2, wherein the display position of the tactile sense indicator moves according to the sliding of the tactile sense request position. 4. The tactile sense presentation system according to claim 3 , wherein the display position of the tactile sense indicator moves according to the tactile sense of each slide position asynchronously with the sliding of the tactile sense request position. 5. The tactile sensation presentation system according to claim 1 , wherein the shape of said tactile indicator changes according to the tactile sensation. 6. The tactile sense presentation system according to any one of claims 1 to 5 , wherein the display color of said tactile sense indicator changes according to the tactile sense. 7. The tactile sense presentation system according to any one of claims 1 to 6 , wherein the tactile sense indicator vibrates at an amplitude and/or frequency corresponding to the tactile sense. 8. The tactile sensation presentation system according to claim 7 , wherein the amplitude and/or frequency varies according to the distance from the tactile sensation requesting position. The display comprises a sensor that detects a touch operation,
the tactile sensation requesting operation is a touch operation on the display;
9. The tactile sensation presentation system according to any one of claims 1 to 8 , wherein the size of said tactile indicator changes according to the tactile sensation and pressing pressure of the touch operation. The video is each frame image of a moving image,
10. The method according to any one of claims 1 to 9 , further comprising means for correcting the tactile information generated based on the current frame image based on each tactile information generated based on each frame image before the previous one. The tactile presentation system described. The correcting means corrects the type of the tactile sensation requested position of the current frame by a weighted linear sum based on the distance between the tactile sensation requested position and each position of each frame image before the previous time and the type thereof. The tactile sensation presentation system according to claim 10 . 12. The tactile sensation presentation system according to claim 11 , wherein said means for modifying creates a type histogram for each frame, and does not apply said modification between frames in which a change in the type histogram exceeds a predetermined threshold. The display comprises a sensor that detects a touch operation,
the tactile sensation requesting operation is a touch operation on the display;
further comprising means for detecting how the display is touched based on the touch operation;
13. The tactile sense presentation system according to any one of claims 1 to 12 , wherein the means for displaying the tactile sense indicator displays the tactile sense indicator based on the acquired tactile sense parameters and the manner of touch. In a tactile sensation presentation method in which a computer visually presents the tactile sensation of an object displayed on a display,
Get an image of an object,
Storing tactile parameters in advance for each type of object,
identify the type and position of the object displayed on the display;
generating tactile information that associates each position on the display with a tactile parameter based on the result of identifying the type and position of the object and the tactile parameter previously stored for each type of object;
Detects the tactile sensation request position based on the tactile sensation request operation on the display,
acquiring a tactile parameter corresponding to the tactile sensation requested position based on the tactile sensation information;
displaying a tactile indicator that visually expresses the tactile sensation at the tactile sensation requested position based on the acquired tactile sensation parameter;
A tactile sensation presentation method, characterized by displaying a plurality of the tactile sensation indicators over the entire background area of the object . In a tactile sensation presentation program that visually presents the tactile sensation of an object displayed on a display,
a procedure for acquiring an image of an object;
identifying the type and location of an object displayed on the display;
a step of generating tactile information that associates each position on the display with a tactile parameter, based on the result of identification of the type and position of the object and tactile parameters stored in advance for each type of object;
a procedure for detecting a tactile sensation requesting position based on a tactile sensation requesting operation on a display;
a procedure for acquiring a tactile parameter corresponding to the tactile sensation requested position based on the tactile sensation information;
a step of displaying a tactile indicator that visually expresses the tactile sensation at the tactile sensation requested position based on the acquired tactile sensation parameter;
on the computer, and
A tactile sensation presentation program characterized by displaying a plurality of the tactile sensation indicators over the entire background area of the object .

Images