JP7115061B2 - Notation conversion device, conversion display device, conversion display system, control method and recording medium - Google Patents

Description

The present invention relates to a notation conversion technology that can be suitably used in notation methods including language notation such as Japanese and English, dialect notation such as standard language and dialect, and character display such as kanji, katakana, and hiragana. The present invention relates to a technique for converting data written in one notation system into notation in another notation system.

2. Description of the Related Art Image forming apparatuses such as printers, copiers, multi-function machines, etc. are provided with an operation panel for receiving various settings. The image forming apparatus disclosed in Patent Document 1 receives personal information data including the nationality of the user from a mobile terminal via short-range wireless communication, determines the official language of the user based on the received nationality, and sets the display language. set. Accordingly, the user can operate the image forming apparatus using the operation panel displayed using a language that the user can understand.

However, assuming that a plurality of users using different languages jointly operate the image forming apparatus, the display language of the operation panel is set according to the first language used by the first user. If so, the first user can understand the display contents of the operation panel and can operate the image forming apparatus, but the second user who uses the second language will not understand the first language. In this case, the contents displayed on the operation panel cannot be understood, and the operation of the image forming apparatus becomes difficult.

In order to solve this problem, according to Patent Document 2, the user wears a transmissive head-mounted image display device, and the user is superimposed on the scenery seen through the image display device and displayed on the image display device. You can visually recognize the image. When the user sees an English character string through this video display device, the character string is identified as an attention area, and the video display device determines the language to be translated to Japanese based on the user information, The English character string in the attention area is automatically translated into Japanese, and the Japanese character string is displayed so as to overlap the English character string in the attention area within the visual field before translation.

For example, when a first user who understands English sees a Japanese character string through a head-mounted image display device, the image display device displays the character string translated into English from the Japanese character string. . Similarly, when a second user who understands Spanish sees the above Japanese character string through the same video display device, the video display device displays the Japanese character string translated into Spanish. be. Therefore, the first user and the second user can know the meaning of the Japanese character string even if they cannot understand Japanese.

In this way, the problem of Patent Document 1 is solved when a plurality of users using different languages try to operate the device together.

JP 2017-211853 A JP 2014-93050 A

However, according to the technology disclosed in Patent Document 2, the first user does not know where the second user is paying attention, and the second user does not know where the first user is paying attention. I don't know if In this way, since the attention points of the other party are not known to each other, this is one of the factors that hinder communication between a plurality of users when a plurality of users work together.

In the above, we have pointed out problems in language expressions such as Japanese, English, and Spanish. exists.

In view of the above points, the present invention provides a notation conversion device, a conversion display device, a notation conversion device, a conversion display device, An object of the present invention is to provide a conversion display system, a control method, and a recording medium.

In order to achieve the above object, one aspect of the present invention is a notation conversion device that converts first data notated in a first display area on a display surface in a first notation into notation in a second notation, , viewpoint determination means for determining whether or not a plurality of viewpoints of viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display region; Determining means for determining, as a second notation, a notation that has an affinity with a plurality of users who focus their eyes on the first display area, except for the first notation, only when there are multiple viewpoints in the display area. and notation conversion means for converting the first data written in the first display area into second data written in the second notation, and the position of the second display area on the display surface. and output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area. (a) the first notation is an expression in a first language, the second notation is an expression in a second language, and (b) the first notation is an expression in Chinese characters. , the second notation is an expression in katakana, hiragana or romaji; (c) the first notation is an expression in katakana, hiragana or romaji, and the second notation is an expression in kanji , (d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the The first notation is an expression in a dialect, and the second notation is an expression in a standard language .

Further, one aspect of the present invention is a notation conversion device that converts first data notated in a first notation in a first display area on a display surface into notation in a second notation, wherein the display surface is viewpoint determination means for determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users who are gazing are within the first display area; and a plurality of viewpoints within the first display area. Determination means for determining, as a second notation, a notation that has an affinity for a plurality of users who focus their eyes on the first display area, except for the first notation, only when there is a second notation; Notation conversion means for converting the first data written in the display area into second data written in the second notation, and position information indicating the position of the second display area on the display surface. generating means for generating; and output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area; , the notation with affinity to be determined is a notation that is registered in advance as a notation that is familiar to each user .

Here, when there are a plurality of notations with affinity for the plurality of users, the determining means selects one notation with the greatest affinity among the plurality of users, It may be determined as the second notation.

Further, one aspect of the present invention is a notation conversion device that converts first data notated in a first notation in a first display area on a display surface into notation in a second notation, wherein the display surface is viewpoint determination means for determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users who are gazing are within the first display area; and a plurality of viewpoints within the first display area. Determination means for determining, as a second notation, a notation that has an affinity for a plurality of users who focus their eyes on the first display area, except for the first notation, only when there is a second notation; Notation conversion means for converting the first data written in the display area into second data written in the second notation, and position information indicating the position of the second display area on the display surface. generating means for generating; and output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area; , when a plurality of notations having an affinity for the plurality of users are determined as the second notation, the notation conversion means converts the first data to each of the determined notations Characterized by converting.

Here, the second display area is divided into a plurality of partial areas each displaying the converted first data according to the plurality of notations, and the generating means further divides the plurality of partial areas into and the output means may output the partial area information.

Here, the output means sequentially outputs the plurality of converted first data in order to switch and display each of the plurality of converted first data in the second display area; may be

Here, the generating means generates the position information indicating the position of the first display area, and the output means displays the second data in the first display area as the second display area. , and output the position information.

Here, the blank area is further provided with blank determination means for determining whether or not there is a blank area in which data is not displayed and which is larger than the size of the second data in an area around the first display area. is determined to exist, the generating means may generate the position information indicating the position of the second display area within the blank area.

Here, the viewpoint determination means further determines that the number of the viewpoints existing within the third display area is a predetermined number or more among the viewpoints on the display surface of the plurality of users gazing at the display surface. If the number is equal to or greater than a predetermined number, the generating means further generates enlarged range information indicating the range of the enlarged area obtained by enlarging the third display area, and the output means further comprises the The enlarged range information may be output to the display device in order to enlarge and display the third display area.

Here, on the display surface, the second display area exists within a predetermined distance from the first display area, the second display area exists in contact with the first display area, and the first display area exists. The second display area may overlap with the second display area, or the second display area may be included in the first display area.

Here, the generating means further generates the same color pattern information indicating the display color or display pattern in the first display area and the display color or display pattern in the second display area, and outputs The means may further output the color pattern information to the display device for displaying using the display color or the display pattern.

Here, the apparatus may further include detection means for detecting a viewpoint on the display surface of the user who gazes at the display surface.

Here, the detection means includes a camera that obtains image data of the user as a subject, and extracts the standing position of the user and the line-of-sight direction of the user from the image data, and extracts the standing position. and a calculation unit that calculates the viewpoint of the user who gazes at the display surface on the display surface based on the line-of-sight direction.

Here, the display device is provided in a posture with the rectangular display surface facing upward, the user stands around the display device, and the generating means further configures the second data. image data is generated by rotating the second data so that the characters appear erect from the standing position of the user, and the output means outputs the image data as the second data and the display device may display the image data in the second display area.

Further, one aspect of the present invention is a notation conversion device that converts first data notated in a first notation in a first display area on a display surface into notation in a second notation, wherein the display surface is viewpoint determination means for determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users who are gazing are within the first display area; and a plurality of viewpoints within the first display area. Determination means for determining, as a second notation, a notation that has an affinity for a plurality of users who focus their eyes on the first display area, except for the first notation, only when there is a second notation; Notation conversion means for converting the first data written in the display area into second data written in the second notation, and position information indicating the position of the second display area on the display surface. generating means for generating; output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area; and the plurality of users and storage means for storing notation information indicating a notation with affinity for each user, and the determination means acquires the notation information from the storage means, and acquires the notation It is characterized by determining a notation method that is compatible with the plurality of users by using the notation method indicated by the legal information.

Here, the viewpoint judging means acquires the viewpoint from spectacles attached to the user's face and provided with a camera and a calculation unit, and the camera captures the direction of the user's line of sight. and the calculation unit may calculate the position of the object appearing in the center of the image data as the viewpoint.

Here, the viewpoint determining means acquires the viewpoint from the eyeglass-type display device equipped with a camera and worn on the user's face, and the display device captures the direction of the user's line of sight with the camera. image data, and the position of the object appearing in the center of the generated image data may be calculated as the viewpoint.

Further, another aspect of the present invention is a conversion display device that converts and displays first data written in a first notation in a first display area on a display surface into notation in a second notation, display means for displaying the first data in the first display area; viewpoint acquisition means for acquiring respective viewpoints on the display surface of a plurality of users who are gazing at the display surface; Determination means for determining whether or not a plurality of viewpoints exist within the first display area; determining means for determining, as a second notation, a notation that has an affinity with a plurality of users who gaze at the area; notation conversion means for converting into second data notated by; generating means for generating position information indicating the position of the second display area on the display surface; and displaying the second data and the position information on the display means and output means for outputting to, the display means receives the second data and the position information, displays the second data in the second display area, (a) the first notation modalities are expressions in a first language, said second notation is expressions in a second language, (b) said first notation is expressions in kanji, and said second notation is in katakana , hiragana or romaji, (c) the first notation is katakana, hiragana, or romaji, the second notation is kanji, (d) the first notation is an expression in a standard language that is used nationwide as a norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first notation is an expression in a dialect , and the second notation is characterized in that it is an expression in standard language .

Another aspect of the present invention is a conversion display system that converts and displays first data written in a first notation in a first display area on a display surface into a notation in a second notation, A notation conversion device, a display device, and a detection device, wherein the detection device is spectacles worn on the user's face and provided with a camera and a calculation unit, and the camera captures the direction of the user's line of sight. Image data is acquired, the calculation unit calculates the position of an object appearing in the center of the image data as a viewpoint, and the notation conversion device acquires from the detection device and gazes at the display surface. viewpoint determination means for determining whether or not a plurality of viewpoints of the respective viewpoints of the person on the display surface exist within the first display area; , determination means for determining, as a second notation, a notation excluding the first notation that has an affinity with a plurality of users who focus their eyes on the first display area; notation conversion means for converting the first data written in the second notation into second data notated according to the second notation; generating means for generating position information indicating the position of the second display area on the display surface; and output means for outputting the second data and the position information to the display device, the display device having the display surface, and the first data being displayed in the first display area of the display surface. receiving the second data and the location information; displaying the second data in the second display area ; (a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in kanji; the second notation is an expression in katakana, hiragana or romaji; (c) The first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, and (d) the first notation is in a standard language used as a norm nationwide or (e) said first notation is an expression in a dialect and said second notation is in a standard language It is characterized by being an expression .

Another aspect of the present invention is a conversion display system that converts and displays first data written in a first notation in a first display area on a display surface into a notation in a second notation, A notation conversion device, a bulletin board, and a detection display device, wherein the bulletin board has the display surface, the detection display device is attached to the user's face, and is a glasses-type display device equipped with a camera, The camera acquires image data by photographing the line-of-sight direction of the user, calculates the position of an object appearing in the center of the acquired image data as a viewpoint, displays the image data, and the notation conversion device and viewpoint determination for determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users gazing at the display surface obtained from the detection display device are within the first display area. means, and only when a plurality of viewpoints exist within the first display area, a notation method that is compatible with a plurality of users who focus their eyes on the first display area, except for the first notation method. Determination means for determining two notations; Notation conversion means for converting the first data written in the first display area into second data notated in the second notation; , generating means for generating position information indicating the position of the second display area, and output means for outputting the second data and the position information to the detection display device, wherein the detection display device receiving the second data and the location information, and displaying the second data in the second display area indicated by the location information ; (a) the first notation is an expression in a first language; The second notation is an expression in a second language; (b) the first notation is an expression in kanji; the second notation is an expression in katakana, hiragana or romaji; ) The first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, and (d) the first notation is a standard language used as a national norm. and the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first notation is an expression in a dialect, and the second notation is a standard language It is characterized by being an expression by

Another aspect of the present invention is a control method used in a notation conversion device that converts first data notated in a first notation in a first display area on a display surface into notation in a second notation. a viewpoint determination step of determining whether or not a plurality of viewpoints among viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area; Determination of determining, as a second notation, a notation that has an affinity with a plurality of users who focus their eyes on the first display area, except for the first notation, only when a plurality of viewpoints exist within the area. a notation conversion step of converting the first data written in the first display area into second data written in the second notation; a generation step of generating position information indicating a position; and an output step of outputting the second data and the position information to an external display device in order to display the second data in the second display area. (a) the first notation is in a first language; the second notation is in a second language; (b) the first notation is in Chinese characters; , the second notation is an expression in katakana, hiragana or romaji; (c) the first notation is an expression in katakana, hiragana or romaji, and the second notation is an expression in kanji , (d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the The first notation is an expression in a dialect, and the second notation is an expression in a standard language .

Further, another aspect of the present invention is a control computer used in a notation conversion device that converts first data notated in a first notation in a first display area on a display surface into notation in a second notation A computer-readable recording medium recording a program, wherein a plurality of viewpoints among the respective viewpoints of a plurality of users gazing at the display surface on the display surface are stored in the notation conversion device, which is a computer. a visual point determination step of determining whether or not it exists within the first display area; a determination step of determining, as a second notation, a notation that is compatible with a plurality of users; a notation conversion step of converting to second data; a generating step of generating position information indicating the position of the second display area on the display surface; and displaying the second data in the second display area, an output step of outputting the second data and the position information to an external display device; (a) the first notation is an expression in a first language; , the second notation is an expression in a second language; (b) the first notation is an expression in kanji; the second notation is an expression in katakana, hiragana or romaji; c) the first notation is an expression in katakana, hiragana or romaji; the second notation is an expression in kanji; and (d) the first notation is a standard used nationwide as a norm or (e) said first notation is a dialect expression and said second notation is a standard It is characterized by being an expression in words .

According to the above aspect, only when the viewpoints of a plurality of users exist in the first display area where the first data is displayed according to the first notation, the display is compatible with a plurality of users who look at these viewpoints. The second data is displayed in the second display area by converting the first data according to the second notation (excluding the first notation) having the correctness.

By displaying the second data in the second display area in this manner, it can be seen that a plurality of users are paying attention to the first display area. In other words, one user who is paying attention to the first display area has another user who is also paying attention to the first display area. You can know that you are interested in the first data displayed in the area. As a result, communication can be achieved among a plurality of users. In addition, since the second data is displayed in the second display area, even a user who is not familiar with the first notation can understand the meaning of the first data displayed in the first display area. can be done.

1 shows an external perspective view of a translation display device 1 of an embodiment. FIG. Horizontal angles of view 321 to 324 of cameras 12a to 12d provided in the translation display device 1 are shown. 2 is a block diagram showing the internal configuration of the translation display device 1. FIG. An example of the data structure of the display data table 171 is shown. 4 shows the relationship between the text data displayed on the display unit 102 of the translation display device 1 and the standing position of the user. An example of the data structure of the user attribute table 191 and an example of face image data are shown. An example of image data is shown. An example of the data structure of the object information table 121 and an example of object data are shown. An example of the data structure of the parts information table 131 and an example of parts data are shown. An example of the data structure of the user position table 141 is shown. An example of the data structure of the horizontal direction table 156 is shown. An example of the data structure of the line-of-sight information table 151 is shown. An example of the data structure of the viewpoint position table 161 is shown. An example of the data structure of the display area table 181 is shown. An example of the data structure of the translation text data table 201 is shown. (a) Schematic representation of a user's head oriented in three directions. (b) indicates the horizontal orientation of the user's head; (c) Schematic representation of a user's head with three tilts relative to the vertical. (d) Shows the inclination of the user's head from the vertical direction. An example of text data and translation text data displayed on the display unit 102 of the translation display device 1 is shown. (a) shows an example when the first display area 351a and the second display area 352a are in contact with each other. (b) shows an example in which the second display area 352b exists within a predetermined distance from the first display area 351b. (c) An example of a case where a second display area 352c exists overlapping the first display area 351c is shown. (d) An example of a case where the second display area 352d is included in the first display area 351d is shown. (e) Shows an example in which the hatching pattern applied to the background within the first display area 351e and the hatching pattern applied to the background within the second display area 352e match. (f) Shows an example where the dot pattern applied to the background in the first display area 351f and the dot pattern applied to the background in the second display area 352f match. (g) An example in which the first display area 351g and the second display area 352g are linked by the line image 353 is shown. (h) Shows an example in which the display pattern of the frame of the first display area 351h and the display pattern of the frame of the second display area 352h match. 4 is a flow chart showing the operation of the translation display device 1; Continue to FIG. 4 is a flow chart showing the operation of the translation display device 1; Continued from FIG. (a) shows the relationship between the text data displayed on the display unit 102 of the translation display device 1 as Modification (1) and the standing position of the user. (b) shows an example of text data and translation text data displayed in a large size on the display unit 102 of the translation display device 1; FIG. 10 shows an example of the relationship between the text data displayed on the display unit 102 of the translation display device 1 and the user's standing position, and the text data and translation text data displayed on the display unit 102 as Modified Example (2). A schematic configuration of a translation display system 501 as a modified example (3) is shown. (a) shows a schematic configuration of a translation display system 531 as a modified example (4); (b) shows an example of an image displayed on the display unit 533b of the glasses 533;

1 Embodiment A translation display device 1 as one embodiment of the present invention will be described.

1.1 Translation display device 1
FIG. 1 shows an external perspective view of a translation display device 1. As shown in FIG.

The translation display device 1 is a table-type display device, as shown in this figure. A rectangular display unit 102 is incorporated in the top plate of the translation display device 1, and cameras 12a, 12b, 12c, and 12d are installed at four corners of the top plate, respectively. A main body of the translation display device 1 is supported by legs 11 . In addition, scales 15a, 15b, 15c, and 15d are provided at equal intervals, for example, at intervals of 5 cm, on the four peripheries of the top plate.

The display unit 102 is provided with a rectangular display surface facing upward. The user stands around the translation display device 1 having the display unit 102 .

In this specification, the origin is set at one vertex 102a of the rectangular display section 102, the X axis and the Y axis are set along the short side 102b and the long side 102c of the rectangular display section 102, and the Z axis is set in the vertical direction. Set axis. Also, the space in which the translation display device 1 exists is specified by the XYZ coordinates, and the display surface of the display unit 102 is specified by the XY coordinates.

FIG. 2 shows horizontal angles of view 321-324 of cameras 12a-12d, respectively.

The camera 12a is installed such that its optical axis is oriented approximately 22.5 degrees upward from the horizontal plane and toward a diagonal corner, and its photographing direction is fixed. The horizontal angle of view 321 of the camera 12a is, for example, approximately 90 degrees, and the vertical angle of view is, for example, approximately 45 degrees. Cameras 12b, 12c and 12d are similar to camera 12a.

As shown in FIG. 2, the imaging range of the cameras 12a to 12d covers almost all the surroundings of the translation display device 1, so that the cameras 12a to 12d can capture the image of a user standing around the translation display device 1, particularly the user. face can be photographed.

1.2 Internal Configuration of Translation Display Device 1 FIG. 3 is a block diagram showing the internal configuration of the translation display device 1 .

As shown in this figure, the translation display device 1 includes cameras 12a to 12d, a main control unit 101, a display unit 102, an output unit 103, a storage unit 104, input units 105a to 105d, a face recognition processing unit 106, a position calculation unit. 107 , line-of-sight point calculation unit 108 , display area detection unit 109 , display control unit 110 , translation unit 111 and synthesis unit 229 .

Here, the main control unit 101, part of the storage unit 104, the display control unit 110, and the translation unit 111 convert data written in a first language (first writing system) into one display area on the display surface. It constitutes a translation device (notation conversion device) that translates (converts) into notation in two languages (second notation). Here, the first notation is the expression in the first language and the second notation is the expression in the second language. In addition, the cameras 12a to 12d, the main control unit 101, a part of the storage unit 104, the input units 105a to 105d, the face recognition processing unit 106, the position calculation unit 107, the sight line point calculation unit 108, and the display area detection unit 109, It constitutes detection means for detecting the viewpoint on the display surface of the user who gazes at the surface. Each of the cameras 12a-12d obtains image data of the user as a subject. The face recognition processing unit 106, the position calculation unit 107, the line-of-sight point calculation unit 108, and the display area detection unit 109 extract the user's standing position and the user's line-of-sight direction from the image data, and based on the extracted standing position and line-of-sight direction. together constitute a calculation unit for calculating the viewpoint on the display surface of the user who gazes at the display surface.

The translation display device 1 includes a memory that stores a control computer program and a processor that operates according to the computer program. The main control unit 101, the face recognition processing unit 106, the position calculation unit 107, the line-of-sight point calculation unit 108, the display area detection unit 109, the display control unit 110, and the translation unit 111 each operate according to a computer program. fulfill its function.

(1) Storage unit 104
The storage unit 104 is composed of, for example, a hard disk.

The storage unit 104 stores image data 111a, 111b, 111c, 111d, an object information table 121, object data 128, . Line-of-sight information table 151, horizontal direction table 156, viewpoint position table 161, display data table 171, display area table 181, user attribute table 191, face image data 197a, 197b, 197c, ..., translation text data table 201, dictionary It has areas for storing database 211, display image data (FIG. 5), and other data (not shown).

(Display data table 171)
FIG. 4 shows an example of the data structure of the display data table 171. As shown in FIG.

The display data table 171 is a data table for storing text data (first data) displayed by the display unit 102 . The display data table 171 includes a plurality of pieces of display data related information 172 in advance, as shown in this figure.

Each display data related information 172 includes data ID 173, display position 174, width 175, height 176, text data 177 and language 178 used.

Data ID 173 is an identifier for identifying display data related information 172 . A display position 174 indicates the coordinate values (X coordinate, Y coordinate) on the display unit 102 of the lower left point of the first display area where the text data 177 is displayed. Width 175 and height 176 indicate the width and height, respectively, of the first display area displaying text data 177 . Text data 177 indicates the content displayed in the first display area of display unit 102 . Here, it is text data in consideration of ease of translation. The used language 178 indicates the language of the text data 177 .

FIG. 5 shows display image data displayed on the display unit 102 of the translation display device 1 based on the display data table 171 as an example.

Identified by data IDs "K001", "K002", "K003", "K004", . The text data in the display data related information is displayed.

(User attribute table 191)
FIG. 6 shows an example of the data structure of the user attribute table 191. As shown in FIG.

The user attribute table 191 is a data table that holds attributes of each user.

The user attribute table 191 contains a plurality of pieces of user attribute information 192 in advance, as shown in this figure.

Each user attribute information 192 is composed of a user ID 193 , a user name 194 , a language used 195 and a face image ID 196 .

A user ID 193 is an identifier that identifies a user. User name 194 is a name indicating a user. The language used 195 (notation information) is a language that the user can understand. Here, comprehensible means that the user can read (see) and understand the characters of the language, but it may also mean that the user can hear, understand, speak, and write. In this way, the language used 195 indicates a language (notation) that is familiar to each user. The face image ID 196 is an identifier for identifying face image data representing the face image of the user.

(face image data)
The storage unit 104 has an area for storing a plurality of face image data.

FIG. 6 also shows an example of face image data.

The face image data 197a, 197b, 197c, 197d, . . . shown in FIG.・Compatible with

Since the face image data is used for recognizing the user, the face image taken from the front of the user's face is preferable. In addition, a face image taken from the left side of the user's face and a face image taken from the right side of the face may be included.

(image data)
The storage unit 104 has areas for storing image data 111a, 111b, 111c, and 111d (FIG. 3).

The image data 111a, 111b, 111c, and 111d are multi-tone image data generated by shooting with the cameras 12a, 12b, 12c, and 12d, respectively. A plurality of pixels are arranged in a matrix in the image data.

FIG. 7 shows image data 111a generated by photographing with the camera 12a as an example of image data. The image data 111 a includes images 343 a , 343 b , 343 c , and 343 d of a plurality of users standing around the translation display device 1 and a partial image 343 e of the top plate of the translation display device 1 . Since the photographing direction of the camera 12a is fixed, the same portion of the tabletop of the translation display device 1 is always reflected in the image data 111a. Scales 344a, 344b, and 344c attached to the periphery of the tabletop are reflected in the image 343e. Therefore, the standing position of the user represented by the images 343a, 343b, 343c, and 343d can be known from the positional relationship between the scales 344a, 344b, and 344c in the image 343e and the images 343a, 343b, 343c, and 343d.

The image data 111a shown in FIG. 7 shows object data 342a, 342b, 342c, and 342d, which are respective face portions of the user, among images 343a, 343b, 343c, and 343d. The shape of each object data is a rectangle.

Image data 111b, 111c, and 111d generated by shooting with the cameras 12b, 12c, and 12d are similar to the image data 111a.

The image data 111a, 111b, 111c, and 111d are still images, respectively, but may be moving images.

(Object information table 121)
FIG. 8 shows an example of the data structure of the object information table 121. As shown in FIG.

The object information table 121 is a data table for storing object information indicating object data, which is a user's face part captured in image data generated by photographing by the cameras 12a to 12d.

The object information table 121 has areas for storing a plurality of pieces of object information 122, as shown in this figure. Object information 122 corresponds to one piece of object data, and each object information 122 includes camera ID 123 , image ID 124 , object ID 125 , object range 126 and user ID 127 .

The camera ID 123 is an identifier for identifying the camera that generated the image data including the object data. The image ID 124 is an identifier for identifying image data including the object data. The object ID is an identifier for identifying the object data. The object range 126 indicates the coordinates of the lower left point and upper right point of the rectangular object data in the image data. A user ID is an identifier that identifies a user whose face is included in the object data.

In addition, since different cameras may photograph one user, the object information table 121 may contain a plurality of pieces of object information for the same user.

(object data)
The storage unit 104 has areas for storing a plurality of object data 128, . . .

The object data 128, .

FIG. 8 also shows an example of object data.

The object data 128 shown in FIG. 8 corresponds to the object ID "OB0001" of the object information table 121 shown in FIG.

Object data is used to detect the user's standing position and the user's line of sight.

(Parts information table 131)
FIG. 9 shows an example of the data structure of the parts information table 131. As shown in FIG.

The parts information table 131 has areas for storing a plurality of parts information 132 as shown in this figure.

Here, the parts (left eye, right eye, mouth, nose, left ear, right ear, etc.) that make up the object (face) are called parts. Part data, which are partial images of the mouth, nose, left ear, right ear, etc., are cut out. Here, the shape of the parts data is a rectangle.

The parts information 132 corresponds to one part data, and each parts information 132 includes an object ID 133 , a parts ID 134 , a parts range 135 and a parts type 136 .

The object ID 133 is an identifier that identifies object data containing the parts data. The parts ID 134 is an identifier that identifies the parts data. The parts range 135 indicates the coordinates of the lower left point and the upper right point of the rectangular parts data in the object data containing the parts data. The part type 136 indicates the type of part data. That is, the part type 136 indicates whether the part data is left eye, right eye, mouth, nose, left ear, or right ear.

(parts data)
The storage unit 104 has an area for storing a plurality of parts data 137a, .

The parts data 137a, .

FIG. 9 also shows an example of parts data.

The parts data 137a to 137f shown in FIG. 9 respectively correspond to the part IDs "PT0001" to "PT0006" of the object ID "OB0001" in the parts information table 131 of FIG.

Parts data is used to detect the line of sight of the user.

(User position table 141)
FIG. 10 shows an example of the data structure of the user position table 141. As shown in FIG.

The user position table 141 is a data table for storing user position information indicating the standing position of the user around the translation display device 1 .

The user position table 141 has an area for storing a plurality of pieces of user position information 142, as shown in this figure. User location information 142 corresponds to one user.

Each user location information 142 includes user ID 143 and user location 144 .

User ID 143 is an identifier for identifying the user. The user position 144 indicates the position of the central portion of the user's face standing around the translation display device 1 (that is, the user's standing position) in XYZ coordinates.

(Horizontal direction table 156)
FIG. 11 shows an example of the data structure of the horizontal direction table 156. As shown in FIG.

The horizontal direction table 156 is a data table used to calculate the horizontal direction of the user's face.

The horizontal direction table 156 stores in advance a plurality of pieces of horizontal direction information 157 as shown in this figure. Horizontal orientation information 157 corresponds to one horizontal orientation of the user's face.

Each horizontal information 157 includes a ratio 158 and a horizontal angle 159 .

Ratio 158 indicates the position of the center line in percentage between the left crossing position and the right crossing position. Specifically, a ratio of "left 0" indicates that the center line coincides with the left crossing position. The ratio "left 25" indicates that the center line is located 25% from the left crossing position between the left crossing position and the right crossing position. A ratio of "50" indicates that the center line exists at the middle position between the left crossing position and the right crossing position. The ratio "Right 25" indicates that the center line exists at a position 25% from the right crossing position between the left crossing position and the right crossing position. A ratio "Right 0" indicates that the center line coincides with the right intersection location.

Here, the center line, the left crossing position and the right crossing position will be described with reference to FIG. 16(a).

The center line 364a is a straight line passing through the middle point between the left eye and the right eye in the user's head 363a in the vertical direction of the head. A left intersection position 367 is a position where a line 369 connecting both eyes and a left contour line 366a intersect, and a right intersection position 368 is a position where a line 369 connecting both eyes and a right contour line 365a intersect. The left contour line 366a indicates the contour line on the left side of the head 363a, and the right contour line 365a indicates the contour line on the right side of the head 363a.

Here, the distance between the center line and the left crossing position is called the left distance, and the distance between the center line and the right crossing position is called the right distance.

The horizontal angle indicates the horizontal orientation of the user's face in degrees.

For example, in the horizontal direction table 156 shown in FIG. 11, the ratio "left 0" corresponds to the horizontal direction angle "left 90". A ratio of "left 0" indicates that the user's face is facing left by 90 degrees from the front.

Also, the ratio "left 25" corresponds to the horizontal angle "left 45". Here, the ratio "left 25" indicates that the user's face is facing leftward by 45 degrees from the front.

Furthermore, the ratio "50" corresponds to the horizontal angle "0". Here, the ratio "50" indicates that the user's face is facing forward.

The same applies to other cases.

(Line-of-sight information table 151)
FIG. 12 shows an example of the data structure of the line-of-sight information table 151. As shown in FIG.

The line-of-sight information table 151 is a data table for storing line-of-sight information indicating the user's line of sight.

The line-of-sight information table 151 has areas for storing a plurality of pieces of line-of-sight information 152, as shown in this figure. The line-of-sight information 152 corresponds to the line-of-sight of one user.

Each line-of-sight information 152 includes user ID 153 , user location 154 and direction information 155 .

User ID 153 is an identifier for identifying the user. The user position 144 indicates the position of the central portion of the user's face standing around the translation display device 1 in XYZ coordinates. Direction information 155 indicates the direction in which the user's line of sight is directed by a vector (X, Y, Z) from user position 144 .

(Viewpoint position table 161)
FIG. 13 shows an example of the data structure of the viewpoint position table 161. As shown in FIG.

The viewpoint position table 161 is a data table for storing viewpoint position information indicating the user's viewpoint on the display surface of the display unit 102 .

The viewpoint position table 161 has an area for storing a plurality of viewpoint position information 162, as shown in this figure. The viewpoint position information 162 corresponds to the viewpoint of one user.

Each viewpoint position information 162 includes a user ID 163 and a viewpoint position 164 .

User ID 163 is an identifier for identifying the user. The viewpoint position 164 indicates the position of the user's viewpoint on the display surface of the display unit 102 in XY coordinates.

(Display area table 181)
FIG. 14 shows an example of the data structure of the display area table 181. As shown in FIG.

The display area table 181 is a data table for storing display area information indicating a user who pays attention to each first display area displayed by the display unit 102 .

The display area table 181 has areas for storing a plurality of pieces of display area information 182, as shown in this figure. The display area information 182 corresponds to one piece of text data.

There are cases where each display area information 182 includes only the data ID 183 and cases where each display area information 182 includes the data ID 183 and one or more user IDs 184 .

Data ID 183 is an identifier for identifying text data displayed by display unit 102 . A user ID 184 is an identifier that identifies a user.

When the display area information 182 includes only the data ID 183, it indicates that no user is paying attention to the first display area where the text data indicated by the data ID 183 is displayed. Also, when the display area information 182 includes the data ID 183 and one user ID 184, it indicates that one user is paying attention to the first display area where the text data indicated by the data ID 183 is displayed. Furthermore, when the display area information 182 includes the data ID 183 and multiple user IDs 184, it indicates that multiple users are paying attention to the first display area where the text data indicated by the data ID 183 is displayed.

(Translation text data table 201)
FIG. 15 shows an example of the data structure of the translated text data table 201. As shown in FIG.

The translation text data table 201 is a data table for storing translation information indicating translated text (translation text data, second data).

The translation text data table 201 has areas for storing a plurality of pieces of translation information 202, as shown in this figure. The translation information 202 corresponds to one piece of text data.

Each translation information 202 includes data ID 203 , translation text 204 and language 205 used.

A data ID 203 is an identifier for identifying text data displayed by the display unit 102 . The translated text 204 is a character string resulting from translating the text data identified by the data ID 203 in the display data table 171 . The used language 205 indicates the language used for translation.

(Dictionary database 211)
The dictionary database 211 is used by the translation unit 111 and is a database that stores correspondence relationships between terms in a first language and terms in a second language for a plurality of languages.

(2) Cameras 12a-12d and input units 105a-105d
The camera 12a photographs an object, generates image data, and outputs the generated image data to the input unit 105a. The input unit 105a receives image data from the camera 12a and writes the received image data as image data 111a into the storage unit 104 together with the camera ID that identifies the camera 12a.

The cameras 12b to 12d and the input units 105b to 105d are the same as the camera 12a and the input unit 105a. The image data 111b to 111d are written in the storage unit 104 together with the camera ID.

(3) Face recognition processing unit 106
The face recognition processing unit 106 reads each image data together with the camera ID from the storage unit 104, and cuts out object data, which is the user's face portion, from the read image data. Extraction of object data is performed, for example, by recognizing a set of eyes, nose, mouth, and ears in image data. The face recognition processing unit 106 also cuts out the eyes, nose, mouth, and ears as part data. Furthermore, the face recognition processing unit 106 determines whether the part data is the eyes, nose, mouth, or ears for each part data, and specifies the part type.

The face recognition processing unit 106 writes object data and part data cut out from each image data into the storage unit 104 .

The face recognition processing unit 106 also generates an object ID for identifying the cut-out object data, and generates an object range indicating the range of the cut-out object data. Next, the face recognition processing unit 106 writes object information including the camera ID, image ID, object ID, and object range into the object information table 121 .

Further, the face recognition processing unit 106 determines whether or not the characteristic portion of the extracted object data matches any of the characteristic portions of the face image data 197a, 197b, 197c, . to judge whether If the feature portion of any face image data matches, the user ID corresponding to the face image ID that identifies that face image data is extracted from the user attribute table 191 . The face recognition processing unit 106 writes the extracted user ID to the object information table 121. FIG. If the characteristic portion of the extracted object data does not match any characteristic portion of the face image data, no user ID is written in the object information table 121 .

In addition, the face recognition processing unit 106 generates a part ID for identifying part data for each part data cut out from the object data, and generates a part indicating the range of the cut out part data. Next, the face recognition processing unit 106 writes parts information including an object ID, a parts ID, a parts range, and a parts type into the parts information table 131 for each part data.

(4) Position calculator 107
The position calculation unit 107 extracts an object range for each piece of object information from the object information table 121, and calculates the extracted object range and the scales attached to the periphery of the tabletop of the translation display device 1 included in the image data. Then, the user position, which indicates the standing position of the user, is calculated by XYZ coordinates. Note that, as described above, the user's standing position indicates the position of the central portion of the user's face standing around the translation display device 1 . The position calculation unit 107 writes the user position information including the user ID for identifying the user and the calculated user position to the user position table 141 .

(5) Line-of-sight viewpoint calculator 108
(calculating the horizontal orientation of the user's face)
The line-of-sight viewpoint calculation unit 108 calculates the horizontal orientation of the user's face for each user as follows.

FIG. 16(a) shows a schematic diagram of the user's head facing in three directions. That is, it shows a user's head 363a facing left, a user's head 363b facing front, and a user's head 363c facing right.

In the head 363a facing leftward, a center line 364a, which is a straight line passing vertically through the head 363a at the midpoint between the left eye and the right eye, is set, and a left contour line 366a indicating the contour of the left side of the head 363a is set. , a right contour line 365a indicating the right contour of the head 363a is set, the center line 364a is closer to the left contour line 366a than the right contour line 365a.

Also, in the head 363b facing the front, similarly to the head 363a, if a central line 364b, a left contour 366b, and a right contour 365b are set, the central line 364b is divided into a right contour 365b and a left contour 366b. It exists in the position of about the center of .

In the head 363c facing right, if a central line 364c, a left contour 366c, and a right contour 365c are set in the same way as for the head 363a, the central line 364c is closer to the right contour 365c than the left contour 366c. exists in a position close to

In this manner, the line-of-sight viewpoint calculation unit 108 specifically determines whether the center line of the head exists on the side closer to the left contour line or on the side closer to the right contour line. can use the horizontal orientation table 156 to estimate the horizontal orientation of the user's face.

Cameras 12a, 12b, 12c, and 12d are installed at the four corners of the top plate of the translation display device 1, respectively, so that one of the cameras can detect the user's head (especially the left eye, right eye, nose, mouth, etc.). etc.) can be photographed. As a result, as shown in FIG. 16(b), it is possible to estimate the horizontal orientations 362a, 362b, 362c of the heads 361a, 361b, 361c of the users.

The line-of-sight point calculation unit 108 reads object data identified by one object ID from the storage unit 104 for each object ID stored in the object information table 121 . Further, for the object ID, the line-of-sight viewpoint calculation unit 108 extracts, from the parts information table 131, the part ID and the parts range in the parts information for the left eye and the parts ID and the parts range in the parts information for the right eye. Parts data identified by the left eye parts ID and the right eye parts ID are read out from the storage unit 104 .

Next, the line-of-sight viewpoint calculation unit 108 calculates the center line using the parts data and parts range for the left eye, the parts data and parts range for the right eye, and the object data. Also, the line-of-sight viewpoint calculation unit 108 calculates the left contour and the right contour using the object data. Next, the line-of-sight viewpoint calculator 108 uses the object data to calculate the ratio between the left distance between the center line and the left contour line and the right distance between the center line and the right contour line. For example, if the centerline is in the middle of the left and right contours, the ratio is 50%. If the center line coincides with the left contour line, the percentage is left 0%. If the center line coincides with the right contour line, the percentage is right 0%. If the ratio of the left distance between the center line and the left contour to the right distance between the center line and the right contour is 1:3, then the ratio is left 25%. If the ratio of the left distance between the center line and the left contour to the right distance between the center line and the right contour is 3:1, then the ratio is right 25%.

Next, the line-of-sight viewpoint calculator 108 extracts the horizontal direction angle corresponding to the calculated ratio from the horizontal direction table 156 . If the horizontal angle corresponding to the calculated ratio does not exist in horizontal table 156, linear interpolation is used.

In this way, the horizontal orientation of the user's face is calculated.

(Calculation of tilt of user's face from vertical direction)
The line-of-sight point calculation unit 108 calculates, for each user, the tilt of the user's face (head) from the vertical direction as follows.

FIG. 16(c) shows a user's face 375a that is not tilted with respect to the vertical direction, a user's face 375b that is slightly tilted with respect to the vertical direction, and a user's face 375c that is further tilted with respect to the vertical direction. .

In the user's face 375a that is not tilted with respect to the vertical direction, the center line 374a remains substantially vertical. Also, in the user's face 375b slightly tilted with respect to the vertical direction, the center line 374b is slightly tilted from the vertical direction. In addition, in the user's face 375c that is further tilted with respect to the vertical direction, the center line 374c is further tilted from the vertical direction.

Therefore, by using the center line of the user's head, it is possible to know the tilt of the user's face from the vertical direction. That is, as shown in FIG. 16(d), line-of-sight directions 373a and 373b can be known from the inclination of the center lines 372a and 372b of the user's faces 371a and 371b from the vertical direction.

As in the case of calculating the horizontal orientation of the user's face, the line-of-sight point calculation unit 108 uses the parts data and parts range for the left eye, the parts data and parts range for the right eye, and the object data to calculate the center line. calculate.

Next, based on the calculated center line, the line-of-sight viewpoint calculation unit 108 calculates the direction perpendicular to the center line, and uses this as the tilt of the user's face from the vertical direction.

(Generation of line-of-sight information)
For each user, the line-of-sight point calculation unit 108 synthesizes the calculated horizontal orientation of the user's face and the tilt of the user's face from the vertical direction, and obtains direction information (X, Y, Z ).

Next, the line-of-sight point calculation unit 108 calculates, for each user, line-of-sight information including a user ID that identifies the user, user position information written in the user position table 141, and direction information in which the user's line of sight is directed. Write to table 151 .

(Generation of viewpoint information)
The line-of-sight viewpoint calculation unit 108 sequentially reads all pieces of line-of-sight information from the line-of-sight information table 151 . The line-of-sight point calculation unit 108 extends a straight line in the direction indicated by the direction information included in the read-out line-of-sight information from the position indicated by the user position included in the read-out line-of-sight information, and the extended straight line is displayed on the display unit 102. is calculated as the point of view of the user. The line-of-sight viewpoint calculation unit 108 writes viewpoint position information including a user ID for identifying a user and a viewpoint position into a viewpoint position table 161 .

(6) Display area detector 109
The display area detection unit 109 sequentially reads all the display data related information from the display data table 171 and writes the display area information including the data ID in the read display data related information to the display area table 181 . As a result, the display area table 181 includes the same number of display area information as the display data related information included in the display data table 171 .

Next, the display area detection unit 109 sequentially reads all pieces of viewpoint position information from the viewpoint position table 161 . Each time the viewpoint position information is read, the display area detection unit 109 detects the viewpoint position included in the read viewpoint position information according to the display position, width, and height indicated by which display data related information in the display data table 171. Determine if it exists within the indicated first display area. When the viewpoint position exists within the first display area, the user ID corresponding to the viewpoint position is written in the display area information including the same data ID in the display area table 181 . If the viewpoint position does not exist within the first display area, the user ID is not written.

In this way, the user IDs of users who pay attention to the first display area can be grouped for each first display area.

(7) Display control unit 110
The display control unit 110 reads all display data related information from the display data table 171 . Display image data displaying text data included in the display data-related information in the first display area indicated by the display position, width and height included in the display data-related information for each read display data-related information. Generate. Here, the language used included in the display data-related information, that is, the language on which the text data to be displayed is based is called a first language (first notation). Next, display control section 110 outputs the generated display image data to synthesizing section 229 .

In addition, the display control unit 110 (viewpoint determination means) determines whether or not the viewpoints of a plurality of users exist within one first display region for each display region information included in the display region table 181. It is determined whether or not only the viewpoint exists within one first display area, and whether or not the user's viewpoint exists within one first display area. In other words, the display control unit 110 determines whether or not a plurality of viewpoints among the viewpoints on the display surface of the plurality of users who are gazing at the display surface exist within the first display area.

The display control unit 110 (determining means) selects the data ID included in the display area information to be processed only when the viewpoints of a plurality of users exist within one first display area. A language (first language) is acquired from the display data table 171, and the languages used by a plurality of users who focus their eyes on a plurality of viewpoints present in one first display area are stored in the user attribute table, except for the acquired language. 191, and determines at least one of the acquired languages used by the plurality of users as a second language (second notation). That is, the display control unit 110 uses the acquired working language to determine, as the second language, a language (notation) that has an affinity with a plurality of users who focus their eyes on the first display area. Here, the friendly notation determined by the display control unit 110 (determining means) may be a notation registered in advance as a familiar notation for each user. The display control unit 110 controls the translation unit 111 to translate the text data displayed in the first display area into the second language. The display control unit 110 (generating means) generates position information indicating the position of the second display area attached to the first display area on the display surface. Details of the generation of position information will be described later. The display control section 110 (output means) outputs the translation text data and the position information to the synthesizing section 229 .

Here, when the viewpoints of a plurality of users exist within one first display area, the languages used by the plurality of users who focus their gazes on the plurality of viewpoints existing within the first display area are all in the first language. Sometimes the second display area is not displayed.

On the other hand, when only one user's viewpoint exists within one first display area, the display control unit 110 determines that the language used by the user matches the language of the text data displayed in the first display area. or not.

If they do not match, the display control unit 110 controls the translation unit 111 to translate the text data displayed in the first display area into the language used by the user. In addition, the display control unit 110 (generating means) generates position information indicating the position of the second display area attached to the first display area on the display surface. The display control section 110 (output means) outputs the translation text data and the position information to the synthesizing section 229 . In this case, the translation text data displayed in the second display area is displayed in halftone.

If the language used by the user matches the language of the text data displayed in the first display area, the display control unit 110 does not perform any processing.

(8) Translation unit 111
The translation unit 111 (notation conversion means) converts text data written in the first display area into translated text data translated (notated) in a second language.

Specifically, the translation unit 111 translates the text data using the dictionary database 211 in the determined second language under the control of the display control unit 110, and translates the data ID, the translation text which is the translated text data, and the translation information consisting of the language used is written in the translation text data table 201 .

(9) Combiner 229
Synthesis unit 229 receives the display image data from display control unit 110, outputs the received display image data to display unit 102 via output unit 103, and causes display unit 102 to display the display image data.

Also, the synthesizing unit 229 receives the translated text data and the position information from the display control unit 110 . When the translated text data and the position information are received, the previously received display image data is synthesized with the received translation text data at the position indicated by the received position information, and the synthesized display image data is output via the output unit 103. and output to the display unit 102 to display the display image data on the display unit 102 .

(10) Output unit 103 and display unit 102
The output unit 103 receives display image data from the synthesizing unit 229 and outputs the received display image data to the display unit 102 .

The display unit 102 receives the display image data from the output unit 103 and displays the received display image data.

The display image data includes text data by its width and height at the display positions included in the display data table 171 . It also contains translation text data at the position indicated by the position information.

(11) Main control unit 101
The main control unit 101 includes the cameras 12a to 12d, the display unit 102, the output unit 103, the input units 105a to 105d, the face recognition processing unit 106, the position calculation unit 107, the line-of-sight viewpoint calculation unit 108, the display area detection unit 109, and the display control. The unit 110, the translation unit 111, and the synthesizing unit 229 are controlled in an integrated manner.

1.3 Example of Screen Displayed by Display Unit 102 FIG. 17 shows an example of text data and translation text data displayed on the display unit 102 .

As shown in this figure, the first display area 301 of the display unit 102 displays "cause and countermeasure" in Japanese. The first display area 301 has viewpoints 311c, 312c and 313c.

A viewpoint 311c is ahead of the line of sight 311b of the user identified by the user ID "U001" standing at the position 311a. The language used by this user is "English". Also, the viewpoint 312c is beyond the line of sight 312b of the user identified by the user ID "U002" standing at the position 312a. The language used by this user is "Japanese". Furthermore, the viewpoint 313c is beyond the line of sight 313b of the user identified by the user ID "U003" standing at the position 313a. The language used by this user is "Spanish".

Thus, Japanese is displayed in the first display area 301, the viewpoints 311c, 312c, and 313c exist, and the language used by the user looking at the viewpoints 311c and 313c is not the displayed Japanese. , are in English and Spanish respectively, the second display areas 401a and 401b associated with the first display area 301 display "Cause and Action" and "Causa y Medidas" in English and Spanish, respectively. It is

Therefore, the three users who focus their eyes on the first display area 301 pay attention to the first display area 301 because the second display areas 401a and 401b attached to the first display area 301 are displayed. Know the existence of other users. Also, three users who focus their eyes on the first display area 301 can understand the meaning of the character data displayed in the first display area 301 and the accompanying second display areas 401a and 401b. Communication can be achieved between human users.

In addition, “temperature” is displayed in Japanese in the first display area 302 of the display unit 102 . A viewpoint 314 c exists in the first display area 302 .

Viewpoint 314c is beyond the line of sight 314b of the user standing at position 314a and identified by user ID "U004". The language used by this user is "German".

In this way, Japanese is displayed in the first display area 302, the viewpoint 314c is present, and the language used by the user looking at the viewpoint 314c is German, not the displayed Japanese. , a second display area 402 associated with the first display area 302 displays "Temperatur" in German. However, since only one viewpoint 314 c exists in the first display area 302 , “Temperatur” is displayed in halftone in the second display area 402 .

In addition, in the first display area 303 of the display unit 102, "the orbit of the sun" is displayed in Japanese. The first display area 303 has a viewpoint 315c and a viewpoint 316c.

Viewpoint 315c is beyond the line of sight 315b of the user standing at position 315a and identified by user ID "U005". The language used by this user is "English". Also, the viewpoint 316c is beyond the line of sight 316b of the user who stands at the position 316a and is identified by the user ID "U006". The language used by this user is "Japanese".

In this way, Japanese is displayed in the first display area 303, the viewpoints 315c and 316c are present, and the language used by the user looking at the viewpoint 315c is English, not the displayed Japanese. Therefore, in the second display area 403 attached to the first display area 303, "Sun's Career" is displayed in English.

Therefore, two users who focus their eyes on the first display area 303 can see that the first display area 303 and the accompanying second display area 403 are displayed, so that the two users who pay attention to the first display area 303 existence can be known. Also, two users who focus their eyes on the first display area 303 can understand the meaning of the character data displayed in the first display area 303 and the accompanying second display area 403. Communication can be achieved between users.

In addition, “satellite photo” is displayed in Japanese in the first display area 304 of the display unit 102 . A viewpoint does not exist in the first display area 304 . Therefore, there is no display of translated character data.

(Relationship between first display area and second display area)
The positional relationship between the first display area and the second display area attached to the first display area will be described with reference to FIGS. 18(a) to (d).

FIG. 18A shows an example in which the first display area 351a and the second display area 352a are in contact with each other. Thus, one side of the first display area 351a and one side of the second display area 352a may contact each other. Since the first display area 351a and the second display area 352a are in contact, the relationship between the first display area 351a and the second display area 352a is clear.

In this case, the display control unit 110 (generating means) generates position information indicating the position of the second display area that contacts the first display area on the display surface.

FIG. 18(b) shows an example in which the second display area 352b exists within a predetermined distance from the first display area 351b. In this way, the second display area 352b may exist in the vicinity of the first display area 351b. Here, the distance between the first display area 351b and the second display area 352b may be, for example, the distance between the center of gravity of the first display area 351b and the center of gravity of the second display area 352b. Also, the predetermined distance may be, for example, twice the distance between the centers of gravity when the first display area 351b and the second display area 352b are in contact with each other. Since the second display area 352b exists within a predetermined distance from the first display area 351b, the relationship between the first display area 351b and the second display area 352b is clear.

In this case, the display control unit 110 (generating means) generates position information indicating the position of the second display area existing in the vicinity of the first display area on the display surface.

FIG. 18(c) shows an example in which a second display area 352c overlaps the first display area 351c. In this case, it is preferable that the characters displayed in the first display area 351c do not overlap the characters displayed in the second display area 352c. Since the second display area 352c exists overlapping the first display area 351c, the relationship between the first display area 351c and the second display area 352c is clear.

In this case, the display control unit 110 (generating means) generates position information indicating the position of the second display area overlapping the first display area on the display surface.

FIG. 18(d) shows an example in which a second display area 352d is included in the first display area 351d. In this case, it is preferable that the characters displayed in the first display area 351d do not overlap the characters displayed in the second display area 352d. Since the second display area 352d is included in the first display area 351d, the relationship between the first display area 351d and the second display area 352d is clear.

In this case, the display control unit 110 (generating means) generates position information indicating the position of the second display area included in the first display area on the display surface.

Next, the correspondence relationship between the display in the first display area and the display in the second display area will be described with reference to FIGS. 18(e) to (h).

FIG. 18(e) shows an example in which the hatching pattern applied to the background within the first display area 351e and the hatching pattern applied to the background within the second display area 352e match. Furthermore, the display colors of the hatching patterns may match. Since the hatching patterns are the same, the relationship between the first display area 351e and the second display area 352e is clear.

FIG. 18(f) shows an example in which the dot pattern applied to the background within the first display area 351f and the dot pattern applied to the background within the second display area 352f match. Furthermore, the display colors of the dot patterns may match. Since the dot patterns are the same, the relationship between the first display area 351f and the second display area 352f is clear.

FIG. 18G shows an example in which the first display area 351g and the second display area 352g are linked by a line image 353. FIG. As shown in this figure, the link may be made by a line image of a double line, or may be made by a line image such as a dashed line, a dashed line, or a wavy line. Since the first display area 351g and the second display area 352g are thus linked by the line image 353, the relationship between the first display area 351g and the second display area 352g is clear.

FIG. 18(h) shows an example in which the display pattern of the frame of the first display area 351h matches the display pattern of the frame of the second display area 352h. Furthermore, the display colors of the frames may match. Since the frame display patterns are the same, the relationship between the first display area 351h and the second display area 352h is clear.

Also, the display color of the characters in the first display area and the display color of the characters in the second display area may be the same (not shown). In this case, since the display colors of the characters are the same, the relationship between the inside of the first display area and the inside of the second display area is clear.

The display color of the first display area and the display color of the second display area, which has no correspondence with the first display area, should be different colors. Also, the display pattern of the first display area and the display pattern of the second display area, which have no corresponding relationship, are made to be different patterns.

In the above case, the display control unit 110 (generating means) generates the same color pattern information indicating the display color or display pattern in the first display area and the display color or display pattern in the second display area. may In addition, the display control unit 110 (output unit) may output color pattern information to the synthesizing unit 229 in order to perform display using display colors or display patterns.

1.4 Operation The operation of the translation display device 1 will be described with reference to the flow charts shown in FIGS. 19 and 20. FIG.

Under the control of the display control unit 110, the display unit 102 displays the text data of all the display data related information included in the display data table 171 at the display position with the width and height. Here, the language from which the text data to be displayed is based is called the first language (step S101).

Next, the cameras 12a to 12d respectively photograph an object, generate image data, and write the generated image data to the storage unit 104 via the input units 105a to 105d. The face recognition processing unit 106 acquires image data from the storage unit 104 (step S102).

Next, the face recognition processing unit 106 detects the user's face from the acquired image data (step S103). Next, the position calculation unit 107 detects the standing position of the user from the acquired image data (step S104). Next, the face recognition processing unit 106 identifies the user using the user attribute table 191 and the user's face image data (step S105).

Next, the line-of-sight point calculation unit 108 calculates the horizontal orientation of the user's face (step S106), and then calculates the inclination of the user's face from the vertical direction (step S107). Next, the line-of-sight point calculation unit 108 calculates the user's line of sight using the horizontal orientation of the user's face and the tilt of the user's face from the vertical direction (step S108). The user's viewpoint on the display surface is calculated (step S109).

Next, the display control unit 110 repeats steps S111 to S124 for each display area information included in the display area table 181 (steps S110 to S125).

The display control unit 110 determines, for one first display area, whether or not the viewpoints of a plurality of users exist within one first display area. Specifically, the display control unit 110 determines whether or not one piece of display area information includes a plurality of user IDs (step S111).

When multiple user viewpoints exist within one first display area, that is, when one display area information includes multiple user IDs ("YES" in step S111), display control unit 110 The languages used by the plurality of users are acquired from the user attribute table 191, and at least one of the languages used by the acquired users is determined as the second language. However, when determining the second language, the first language is excluded (step S112). Next, the translation unit 111 translates the text data and acquires translation text data (step S113). Next, display control unit 110 sets a second display area in which the translated text data should be displayed so as to accompany the first display area (step S114). Next, the display control unit 110 outputs the translation text data and position information indicating the position of the second display area to the synthesizing unit 229 (step S115). Next, the display unit 102 displays the display image data including the second text data in the second display area (step S116).

When the viewpoints of multiple users do not exist within one first display area, that is, when one display area information does not include multiple user IDs ("NO" in step S111), the display control unit 110 It is determined whether or not only one user's viewpoint exists within one first display area, that is, whether or not one display area information includes only one user ID (step S119).

When only one user's viewpoint exists within one first display area, that is, when one display area information includes only one user ID ("YES" in step S119), the display control unit 110 , the language used by the one user is obtained from the user attribute table 191, and it is determined whether or not the language used (first language) of the display text data matches the language used by the user (step S120). If the language used in the display text data and the language used by the user do not match ("NO" in step S120), the translation unit 111 translates the text data and acquires translated text data (step S121). Control unit 110 sets, for example, a second display area in which the translation text data should be displayed so as to accompany the first display area (step S122). Next, the display control unit 110 outputs the translated text data and position information indicating the position of the second display area to the synthesizing unit 229 (step S123). Next, the display unit 102 displays the display image data including the second text data in the second display area. At this time, the second text data is displayed in halftone in the second display area (step S124).

If the language used (first language) of the display text data matches the language used by the user ("YES" in step S120), display control unit 110 does nothing.

Further, when the user's viewpoint does not exist within one first display area, that is, when one display area information does not include the user ID ("NO" in step S119), the display control unit 110 do not do.

After repeating steps S111 to S124, the display control unit 110 shifts the control to step S102 and repeats the process.

1.5 Summary As described above, according to the above embodiment, only when a plurality of users' viewpoints exist in the first display area in which the text data is displayed in the first language, these viewpoints can be displayed. Text data is converted according to a second language (other than the first language) used by a plurality of users who are watching, and the translated text data is displayed in a second display area attached to the first display area.

By displaying the translated text data in the second display area attached to the first display area in this way, it can be seen that a plurality of users are paying attention to the first display area. In other words, one user who is paying attention to the first display area has another user who is also paying attention to the first display area, and that user, like himself, is also paying attention to the first display area. You can know that you are interested in the text data that is being processed. As a result, communication can be achieved among a plurality of users. Also, since the translated text data is displayed in the second display area, even a user who cannot understand the first language can understand the meaning of the text data displayed in the first display area.

2 Modifications Although the present invention has been described based on the above embodiments, it is not limited to the above embodiments. You may make it show below.

2.1 Modification (1)
In the above embodiment, as shown in FIG. 5, even when the user's viewpoints 311c, 312c, and 313c are gathered in the first display area 301, the size of the first display area 301 is not changed. . However, the invention is not so limited.

When the viewpoints of many users gather within the first display area of the display unit 102, the first display area may be enlarged. Also, the second display area displayed in conjunction with the first display area may be displayed in a large size.

FIG. 21(a) shows the relationship between the text data displayed on the display unit 102 and the standing position of the user. FIG. 21(b) shows an example of text data and translation text data displayed in a large size on the display unit 102 of the translation display device 1. FIG.

As shown in FIG. 21( a ), many users' viewpoints are concentrated on the first display area 301 . On the other hand, the first display areas 302, 303, and 304 are not focused on by the user.

In such a case, the first display area 301 is enlarged to form a first display area 421a as shown in FIG. You can make it bigger.

Moreover, in this case, the first display areas 302, 303, and 304 in FIG. Also, as shown in FIG. 21(b), the first display area 302 in FIG. , 303 and 304 may be modified and displayed as first display areas 422 , 423 and 424 . For example, in FIG. 21(a), the first display areas 302, 303, 304 are long in the Y-axis direction, but in FIG. 21(b), the first display areas 422, 423, 424 are It is long in the X-axis direction.

Specifically, the display control unit 110 (viewpoint determination means) may determine whether or not the number of multiple viewpoints present in the third display area is equal to or greater than a predetermined number. When the number of viewpoints is equal to or greater than a predetermined number, the display control unit 110 (generating means) further generates enlarged position information indicating the position of an enlarged area obtained by enlarging the third display area. The display control unit 110 (output means) further outputs the enlarged position information to the synthesizing unit 229 in order to display the text data in the enlarged area indicated by the enlarged position information.

Further, when the number of viewpoints is equal to or greater than a predetermined number, display control section 110 further generates second enlarged position information indicating the position of an enlarged area of the second display area, and enlarges the translated text data. may The display control unit 110 further synthesizes the second enlargement position information and the enlarged translation text data in order to display the enlarged translation text data in the enlarged second display area indicated by the second enlargement position information. 229. Here, the fact that the number of viewpoints existing in the third display area is equal to or greater than a predetermined number means that, for example, the number of viewpoints existing in the third display area 25% (or 50%, 75%, or 90%) or more of the total number of viewpoints present on the display surface.

As described above, according to the modification (1), when many users' viewpoints gather in the third display area, the third display area is enlarged, and the second display is displayed accompanying the third display area. The area is also enlarged, and the displayed text data and translation text data are also enlarged.

Therefore, the user who pays attention to the third display area can clearly see these characters by the large displayed text data and the translated text data.

2.2 Modification (2)
In the above embodiment, as shown in FIG. 17, each character in the text data displayed in the first display areas 301, 302, 303, 304 is viewed from the positive direction of the X-axis to the negative direction of the X-axis. upright. The translation text data displayed in the second display areas 401a, 401b, 402, and 403 are also the same. However, the invention is not so limited.

Each character in the text data and each character in the translation text data displayed in the first display area and the second display area, respectively, according to the direction of the user's line of sight and the user's standing position, so that these users can easily see The orientation of the text data and the translation text data may be adjusted so that the characters appear erect.

FIG. 22 shows an example of the relationship between the text data and translation text data displayed on the display unit 102 of the translation display device 1 and the standing position of the user, and an example of the text data and translation text data displayed on the display unit 102. .

As in FIG. 5, also in this figure, the first display area 301 of the display unit 102 displays "cause and countermeasure" in Japanese. The first display area 301 has viewpoints 311c, 312c and 313c. The standing position of the user who looks at these viewpoints is the same as in FIG.

Thus, Japanese (first language) is displayed in the first display area 301, the viewpoints 311c, 312c, and 313c are present, and the language used by the user looking at the viewpoints 311c and 313c is displayed. English and Spanish (second language), respectively, instead of Japanese, are displayed in the second display areas 433 and 432 attached to the first display area 301 in the same manner as in FIG. The words are labeled "Cause and Action" and "Causa y Medidas".

However, what is different from FIG. 5 is that in the second display areas 433 and 432, the displayed characters appear erect when viewed from the standing position of the user who focuses his/her line of sight on the viewpoints 311c and 313c. is. Furthermore, in the second display area 431, the character data written in Japanese can be seen upright when viewed from the standing position of the user who looks at the viewpoint 312c.

The same applies to the second display area 434 associated with the first display area 302 and the second display area 435 associated with the first display area 303 .

Specifically, the display control unit 110 (generating means) generates image data in which the translated text data is rotated so that the characters constituting the translated text data can be viewed upright from the user's standing position. Generate. The display control unit 110 (output means) outputs image data. The display unit 102 displays image data obtained by rotating the translated text data in the second display area.

In this manner, the displayed content appears to be upright according to the direction of the user's line of sight and the user's standing position, thereby promoting the user's understanding of the displayed content.

2.3 Modification (3)
In the above embodiment, the images of the user captured by the four cameras 12a, 12b, 12c, and 12d provided at the four corners of the top plate of the translation display device 1 are used to determine the line-of-sight direction of the user and The user's viewpoint on the display surface of the display unit 102 is sought. However, the invention is not so limited.

A translation display system 501 as a modified example (3) is a system that translates text data written in a first language into a second language and displays it in a partial first display area on the display surface. 23, it may consist of a monitor 511, a translation device 512, and glasses 513, 514, 515 worn on the user's face. The monitor 511 and translation device 512 are connected by a cable 516 .

The translation display system 501 is obtained by replacing the structural parts for detecting the user's viewpoint with glasses 513, 514, and 515 in the translation display device 1 according to the embodiment, and replacing the display unit 102 in the translation display device 1 with a monitor 511. be.

For this reason, the translation device 512 is composed of the remaining components of the translation display device 1 according to the embodiment, excluding the structural portion for detecting the viewpoint of the user and the display unit 102 .

The glasses 513 communicate with the translation device 512 by short-range wireless communication. The glasses 514 and 515 also communicate with the translation device 512 by short-range wireless, like the glasses 513 .

A camera 513 a that captures the line-of-sight direction of the user wearing the glasses 513 and a calculator are arranged in the frame portion of the glasses 513 . The camera 513a captures the line-of-sight direction of the user and generates image data. Output. Glasses 514 and 515 also have the same structure as glasses 513 .

Translation device 512 acquires, from glasses 513, 514, and 515, respective viewpoints on the display surface of a plurality of users who are gazing at the display surface of monitor 511, and among the acquired viewpoints, a plurality of viewpoints are displayed as the first display. Only when multiple viewpoints exist within the first display area, multiple users who focus their eyes on the multiple viewpoints except for the first language understand each of the multiple viewpoints. At least one of the languages is determined as a second language, and the text data written in the first display area is related to the semantic content of the text data and translated text expressed in the second language. It translates the data into data, generates position information indicating the position of the second display area attached to the first display area on the display surface, and outputs the translated text data and the position information to the monitor 511 .

The monitor 511 has a display surface, displays the text data in a first display area of the display surface, receives the translated text data and position information, and displays the translated text data in a second display area indicated by the position information. do.

(summary)
As described above, according to the modified example (3), by displaying the translated text data in the second display area attached to the first display area, a plurality of users are paying attention to the first display area. I understand. In other words, one user who is paying attention to the first display area has another user who is also paying attention to the first display area, and that user, like himself, is also paying attention to the first display area. You can know that you are interested in the text data that is being processed.

As a result, it is possible to facilitate communication between a plurality of users who focus their eyes on the first display area.

2.4 Modification (4)
In the above embodiment, the images of the user captured by the four cameras 12a, 12b, 12c, and 12d provided at the four corners of the top plate of the translation display device 1 are used to determine the line-of-sight direction of the user and The user's viewpoint on the display surface of the display unit 102 is sought. However, the invention is not so limited.

As a modified example (4), a translation display system 531, whose schematic configuration is shown in FIG. This system is composed of a whiteboard 532, a translation device 536, and glasses 533, 534, and 535 worn on the user's face.

A translation display system 531 is obtained by replacing the structural portion for detecting the viewpoint of the user and the display unit 102 in the translation display device 1 according to the embodiment with glasses 533 , 534 , and 535 .

For this reason, the translation device 536 is composed of the remaining configuration parts of the translation display device 1 according to the embodiment except for the structural part for detecting the user's viewpoint and the display unit 102 .

The glasses 533 communicate with the translation device 536 by short-range wireless communication. The glasses 534 and 535 also communicate with the translation device 536 by short-range wireless, like the glasses 533 .

Whiteboard 532 has a display surface.

The glasses 533 are a glasses-type display device that is worn on the user's face and equipped with a camera 533a. The camera 533a captures the line-of-sight direction of the user to generate image data, the position of the object appearing in the center of the generated image data is calculated as a viewpoint, and the calculated viewpoint is sent to the translation device 536 via short-range wireless communication. send to. Also, the glasses 533 display image data obtained by photographing.

Glasses 534 and 535 also have the same configuration as glasses 533 .

Translation device 536 acquires, from glasses 533, 534, and 535, respective viewpoints on the display surface of whiteboard 532 of a plurality of users who are gazing at the display surface. It is determined whether or not it exists within the display area, and only when multiple viewpoints exist within the first display area, multiple users who focus their eyes on the multiple viewpoints except for the first language understand each of the multiple viewpoints. Determine at least one of the languages as the second language, translate the text data written in the first display area into the second language method, and attach it to the first display area on the display surface Positional information indicating the position of the second display area is generated, and the translation text data and the positional information are transmitted to the glasses 533, 534, 535 by short-range wireless communication.

The glasses 533, 534, and 535 receive the translation text data and the position information indicating the position of the second display area from the translation device 536 by short-range wireless communication, and display the received position information in the second display area indicated by the received position information. display translated text data.

(summary)
As described above, according to the modified example (4), by displaying the translated text data in the second display area attached to the first display area, a plurality of users are paying attention to the first display area. I understand. In other words, one user who is paying attention to the first display area has another user who is also paying attention to the first display area, and that user, like himself, is also paying attention to the first display area. You can know that you are interested in the text data that is being processed.

As a result, it is possible to facilitate communication between a plurality of users who focus their eyes on the first display area.

2.5 Other Modifications (1) In the above embodiment, the display unit 102 displays both text data and translation text data. That is, as shown in FIG. 17, the display unit 102 displays the text data in the first display area 301, and displays the translated text data in the second display areas 401a and 401b attached to the first display area 301. there is However, it is not limited to this.

The display unit 102 may display the translated text data in the first display area 301 instead of the text data.

Specifically, the display control unit 110 (generating means) generates position information indicating the position of the first display area, and the display control unit 110 (output means) outputs the position information to the first display area as the second display area. The generated location information may be output to display the translated text data.

When the viewpoints of multiple users exist in the first display area, if none of the users who focus their eyes on these viewpoints uses the first language, the display surface can be used effectively. can be done.

(2) In the above embodiment, as shown in FIG. 17, the display unit 102 displays text data in the first display area 301, and displays text data in the second display areas 401a and 401b attached to the first display area 301. Displaying translated text data. However, it is not limited to this.

The display control unit 110 (blank determination means) may determine whether or not there is a blank area large enough to display the translation text data around the first display area. In other words, it is determined whether or not there is a blank area that is larger than the display size of the translation text data and in which no characters are displayed. When determining that a blank area exists, the display control unit 110 (generating means) generates position information indicating the position of the second display area in the blank area, and the display control unit 110 causes the display unit 102 to display , the translated character data may be displayed in a blank area around the first display area.

On the other hand, if the display control unit 110 determines that there is no blank area large enough to display the translation text data around the first display area, the display size of the translation text data is slightly reduced and then displayed. , for example, after reducing its size by, for example, 5% to 10% in area, again, whether there is a blank area large enough to display the reduced size translation text data. You can judge. If there is a blank area, the translation character data reduced in size may be displayed in the blank area.

Further, when the display control unit 110 determines that there is no blank area large enough to display the translation text data around the first display area, for example, the first display area may be replaced with the first display area. After moving leftward or rightward by a length of about 5% to 10% of the width of the display area, or moving the first display area to 5% to 10% of the height of the first display area It may be determined again whether or not there is a blank area after moving upward or downward by a length of about %. After moving the display area, if there is a blank area, the translation text data may be displayed in the blank area.

By arranging the translation text data in a blank area where no characters are displayed in this way, the translation text data does not cover other necessary characters.

(3) In the above embodiments and modifications, the text data expressed in the first language (first notation) is displayed in the first display area, and the text data is displayed in the second language (second notation). , and the translated text data is displayed in the second display area. However, the invention is not so limited.

The standard language that is used nationwide as a norm may be regarded as one language, and the dialects that have developed differently depending on the region may be regarded as another language.

The first notation may be an expression in a standard language that is used nationwide as a norm, and the second notation may be an expression in a dialect that has developed differently depending on the region.

In other words, the first language may be a standard language and the second language may be a dialect. The text data expressed in the standard language, which is the first language, may be displayed in the first display area, the text data may be translated into the dialect, which is the second language, and the translated text data may be displayed in the second display area. .

Alternatively, the first notation may be a dialect expression, and the second notation may be a standard language expression.

In other words, the first language may be a dialect and the second language may be a standard language. Text data expressed in a dialect, which is the first language, may be displayed in the first display area, the text data may be translated into standard Japanese, which is the second language, and the translated text data may be displayed in the second display area. .

According to this method, communication can be achieved between a person who understands the standard language and a person who understands the dialect.

(4) In the above embodiments and modifications, the text data expressed in the first language (first notation) is displayed in the first display area, and the text data is displayed in the second language (second notation). , and the translated text data is displayed in the second display area. However, the invention is not so limited.

The first notation may be a representation in kanji, and the second notation may be a representation in katakana, hiragana, or romaji.

At this time, the text data written in kanji, which is the first notation, is displayed in the first display area, the text data is converted into katakana (or hiragana, or romaji), which is the second notation, and converted text data may be displayed in the second display area.

Alternatively, the first notation may be expressed in katakana, hiragana, or romaji, and the second notation may be expressed in kanji.

At this time, text data written in katakana (or hiragana or romaji), which is the first notation, is displayed in the first display area, the text data is converted into kanji, which is the second notation, and converted text data may be displayed in the second display area.

According to this method, communication can be achieved between a person who understands kanji and a person who understands katakana, hiragana or romaji.

(5) In the above embodiments and modifications, the text data expressed in the first language (first notation) is displayed in the first display area, and the text data is displayed in the second language (second notation). , and the translated text data is displayed in the second display area. However, the invention is not so limited.

The first notation may be an expression using a symbol mark symbolizing a predetermined idea, and the second notation method may be an expression using characters to express the idea symbolized by the symbol mark. Also, the second notation may be a notation of a description of the symbol mark. Here, the symbol mark may be a figure symbolizing a predetermined idea.

The symbol mark expressed by the first notation method is displayed in the first display area, the predetermined idea symbolized by the symbol mark is expressed by the second notation method as an explanation, and the explanation is displayed in the second display area. You may As the explanation, translation text data translated into each language may be used.

Symbol marks are, for example, company emblems, crests, insignias, national emblems, family crest symbols, logos, seal impressions, and figures that symbolize individuals, families, companies, organizations, and the like. In this case, the explanation is a character representation of a name or the like indicating an individual, a family, a company, an organization, or the like.

Also, the symbol mark is, for example, a map symbol. Specific examples are map symbols indicating shrines, castle ruins, courts, and lighthouses, and figures indicating railroad (shinkansen) routes. Further, specific examples of explanatory texts are "shrine", "castle ruins", "court", "lighthouse" and "shinkansen", respectively. Furthermore, English, Spanish, and French translations may be added to these explanations.

Also, the symbol mark is, for example, a symbol of an electric circuit. Examples are symbols for transformers, resistors, and transistors. Further, specific examples of explanatory texts are "transformer", "resistor", and "transistor", respectively. Furthermore, English, Spanish, and French translations may be added to these explanations.

(6) In the above embodiments and modifications, the data represented by the first notation is displayed in the first display area, the data is converted by the second notation, and the converted data is converted to the second notation. displayed in the display area. Thus, there is a one-to-one relationship between data and transformed data. However, the invention is not so limited.

A symbol indicating a transistor may be displayed in the first display area, and a description "transistor" and a graph showing the electrical characteristics and temperature characteristics of the transistor may be displayed in the second display area as a detailed description. .

In addition, the map symbol indicating the shrine is displayed in the first display area, and the descriptive text "Shrine" and the detailed description such as the origin, history, and worship object of the shrine are displayed in the second display area. good too.

Also, in the first display area, a figure indicating the route of the Shinkansen (railway) is displayed, and in the second display area, "Shinkansen" is displayed as an explanation and the timetable and operation status of the train are displayed as detailed explanation. good too.

In this way, the first notation is an expression using a symbol mark, which is a figure that symbolizes a given idea, and the second notation method is an expression using characters to express the idea symbolized by the symbol mark. good. Also, the first notation may be based on simple expressions, and the second notation may be based on expressions explaining the simple expressions in detail. Here, the expressions that explain the simple expressions in detail may be in the form of letters or figures containing letters.

(7) In the above embodiment and modifications, when there is only one viewpoint in the first display area, the translation text data is displayed in halftones in the second display area attached to the first display area. It is supposed to be displayed. However, the invention is not so limited.

If only one viewpoint exists in the first display area, the second display area itself may not be displayed.

(8) In the above embodiments and modifications, the user's viewpoint is assumed to be fixed at one point. However, the invention is not so limited.

When the user focuses on the first display area, it is considered that the user's viewpoint is not fixed at one point but moves irregularly inside and outside the first display area. In such a case, if the user's point of view is inside the first display area within a certain period of time (for example, 5 seconds), for example, within 80% of the period, the user's point of view is within the first display It may be considered to exist inside the region.

(9) In the above embodiment, the user attribute table 191 (FIG. 6) stores the language used by the user in association with the user. are getting However, the invention is not so limited.

If the language used by the user can be recognized from the face image of the user captured by a camera, such a method may be used.

(10) In the above embodiment, the display control unit 110 (determining means) selects from the user attribute table 191 the languages used by a plurality of users who gaze at a plurality of viewpoints existing within one first display area. At least one of the languages used by the acquired users is determined as a second language (second notation).

Here, when there are a plurality of notation methods that are compatible with a plurality of users who focus their eyes on the first display area, one notation method that is most compatible with the plurality of users is selected as the first notation method. It may be determined as a binary notation.

In addition, priority is set for each language used, and if there are multiple notations that are compatible with multiple users who focus their eyes on the first display area, the language used by these users is the highest A preferred language may be determined as the second notation.

Further, when the display control unit 110 (determination means) determines a plurality of notation methods that are compatible with a plurality of users as the second notation method, the translation unit 111 (notation conversion unit) converts the text data (the one data) may be translated (converted) into a plurality of determined languages (notations). In this case, multiple translation text data are generated.

In this way, when a plurality of pieces of translated text data are generated, the second display area may be divided into a plurality of partial areas for displaying the translated text data according to the determined notation methods. . The display control unit 110 (generating means) may generate partial area information indicating a plurality of partial areas. The display control unit 110 (output means) may output the partial area information. The display unit 102 displays the translated text data in each of the plurality of partial areas within the second display area.

Further, when a plurality of pieces of translated text data are generated as described above, the display control unit 110 (output means) switches and displays each of the plurality of pieces of translated text data in one second display area. , to sequentially output a plurality of translation text data. The display unit 102 displays a plurality of translation text data while switching them in one second display area.

(11) As noted above, each device is a computer system with a microprocessor and memory. The memory stores a control computer program, and the microprocessor may operate according to the control computer program.

Here, the computer program for control is constructed by combining a plurality of instruction codes indicating instructions to the computer in order to achieve a predetermined function.

Also, the control computer program may be recorded on a computer-readable recording medium such as a flexible disk, hard disk, optical disk, or semiconductor memory.

(12) The above embodiments and modifications may be combined.

In the notation conversion device according to the present invention, when the second data is displayed in the second display area attached to the first display area, a plurality of users who focus their eyes on the first display area can Knowing the presence of users who are paying attention to, it is possible to communicate with users who understand different notations, and convert data notated in the first notation to notation in the second notation. As a technology, it is useful.

1 translation display device 11 legs 12a-12d camera 101 main control unit 102 display unit 103 output unit 104 storage unit 105a-105d input unit 106 face recognition processing unit 107 position calculation unit 108 line-of-sight viewpoint calculation unit 109 display area detection unit 110 display Control unit 111 Translation unit 229 Synthesis unit 501 Translation display system 511 Monitor 512 Translation device 513-515 Glasses 513a-515a Camera 516 Cable 531 Translation display system 532 Whiteboard 533-535 Glasses 533a-535a Camera 533b-535b Display unit 536 Translation device

Claims (22)

A notation conversion device for converting first data notated in a first notation in a first display area on a display surface into notation in a second notation,
viewpoint determination means for determining whether or not a plurality of viewpoints among viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
Output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area ,
(a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in Chinese characters; The second notation is an expression in katakana, hiragana or romaji, (c) the first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, ( d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first The notation is an expression in a dialect, and the second notation is an expression in a standard language
A notation conversion device characterized by: A notation conversion device for converting first data notated in a first notation in a first display area on a display surface into notation in a second notation,
viewpoint determination means for determining whether or not a plurality of viewpoints among viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area;
with
The notation conversion apparatus, wherein the notation with affinity determined by the determining means is a notation registered in advance as a notation familiar to each user. When there are a plurality of notation methods with affinity for the plurality of users, the determining means selects one notation method with the greatest affinity for the plurality of users from among the plurality of users as the second method. 2. The notation conversion device according to claim 1, wherein the notation is determined as a notation. A notation conversion device for converting first data notated in a first notation in a first display area on a display surface into notation in a second notation,
viewpoint determination means for determining whether or not a plurality of viewpoints among viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area;
with
When a plurality of notations having an affinity for the plurality of users are determined by the determination means as the second notation, the notation conversion means converts the first data to the determined plurality of notations, respectively. A notation conversion device characterized by converting according to the notation of . The second display area is divided into a plurality of partial areas each displaying the converted first data according to the plurality of notations,
The generating means further generates partial area information indicating the plurality of partial areas,
5. The notation conversion apparatus according to claim 4 , wherein said output means further outputs said partial area information. The output means sequentially outputs the plurality of converted first data in order to switch and display each of the plurality of converted first data in the second display area. 5. The notation conversion device according to claim 4 . The generating means generates the position information indicating the position of the first display area,
2. The notation conversion apparatus according to claim 1, wherein said output means outputs said position information in order to display said second data in said first display area as said second display area. Furthermore, blank judgment means for judging whether or not there is a blank area larger than the size of the second data and in which data is not displayed exists in the area around the first display area,
2. The notation conversion according to claim 1, wherein when it is determined that a blank area exists, the generating means generates the position information indicating the position of the second display area within the blank area. Device. The viewpoint determining means further determines whether or not the number of viewpoints present in the third display area among the respective viewpoints on the display surface of the plurality of users gazing at the display surface is equal to or greater than a predetermined number. to judge
The generating means further generates expansion range information indicating a range of the expansion area obtained by expanding the third display area when the number is equal to or greater than a predetermined number,
2. The notation conversion apparatus according to claim 1, wherein said output means further outputs said enlarged range information to said display device in order to enlarge and display said third display area. On the display surface, the second display area exists within a predetermined distance from the first display area, the second display area exists in contact with the first display area, and overlaps the first display area. 2. The notation conversion device according to claim 1, wherein the second display area exists within the second display area, or the second display area exists while being included in the first display area. The generating means further
generating the same color pattern information indicating the display color or display pattern in the first display area and the display color or display pattern in the second display area;
2. The notation conversion apparatus according to claim 1, wherein said output means further outputs said color pattern information to said display device in order to display using said display color or said display pattern. 2. The notation conversion apparatus according to claim 1, further comprising detection means for detecting a viewpoint on said display surface of a user who gazes at said display surface. The detection means is
a camera that acquires image data of the user as a subject;
extracting the user's standing position and the user's line-of-sight direction from the image data; 13. The notation conversion device according to claim 12 , further comprising a calculation unit that calculates the following. The display device is provided in a posture with the rectangular display surface facing upward,
The user stands around the display device,
The generation means further generates image data arranged by rotating the second data so that the characters constituting the second data appear erect from the standing position of the user,
The output means outputs the image data as the second data,
The notation conversion device according to claim 1, wherein the display device displays the image data in the second display area. A notation conversion device for converting first data notated in a first notation in a first display area on a display surface into notation in a second notation,
viewpoint determination means for determining whether or not a plurality of viewpoints among viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
output means for outputting the second data and the position information to an external display device in order to display the second data in the second display area;
storage means for storing notation information indicating a notation that is compatible with each of the plurality of users;
The determination means acquires the notation information from the storage means, and determines a notation that is compatible with the plurality of users by using the notation indicated by the acquired notation information. A notation conversion device. The viewpoint determination means acquires the viewpoint from glasses attached to the user's face and provided with a camera and a calculation unit and outside the notation conversion device,
The camera captures a line-of-sight direction of the user to generate image data,
2. The notation conversion device according to claim 1, wherein the calculation unit calculates the position of an object appearing in the center of the image data as a viewpoint. The viewpoint determination means acquires the viewpoint from the eyeglass-type display device equipped with a camera and worn on the user's face,
2. The display device captures a viewing direction of the user with the camera to generate image data, and calculates the position of an object appearing in the center of the generated image data as a viewpoint. The notation conversion device described in . A conversion display device for converting first data written in a first notation in a first display area on a display surface into notation in a second notation and displaying the converted data,
display means for displaying the first data in the first display area;
viewpoint acquisition means for acquiring respective viewpoints on the display surface of a plurality of users gazing at the display surface;
determination means for determining whether or not a plurality of viewpoints among the acquired viewpoints are present within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
an output means for outputting the second data and the position information to the display means;
the display means receives the second data and the position information and displays the second data in the second display area ;
(a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in Chinese characters; The second notation is an expression in katakana, hiragana or romaji, (c) the first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, ( d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first The notation is an expression in a dialect, and the second notation is an expression in a standard language
A conversion display device characterized by : A conversion display system for converting first data written in a first notation in a first display area on a display surface into notation in a second notation and displaying the data,
Equipped with a notation conversion device, a display device and a detection device,
The detection device is glasses that are worn on the user's face and include a camera and a calculation unit,
The camera acquires image data by photographing the line-of-sight direction of the user,
The calculation unit calculates the position of an object appearing in the center of the image data as a viewpoint,
The notation conversion device is
viewpoint determination means for determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users gazing at the display surface obtained from the detection device are within the first display area; ,
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
an output means for outputting the second data and the position information to the display device;
The display device has the display surface, displays the first data in the first display area of the display surface, receives the second data and the position information, and displays the display the second data ,
(a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in Chinese characters; The second notation is an expression in katakana, hiragana or romaji, (c) the first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, ( d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first The notation is an expression in a dialect, and the second notation is an expression in a standard language
A conversion display system characterized by: A conversion display system for converting first data written in a first notation in a first display area on a display surface into notation in a second notation and displaying the data,
Equipped with a notation conversion device, a bulletin board and a detection display device,
The bulletin board has the display surface,
The detection display device is a spectacles-type display device that is worn on the face of a user and is equipped with a camera. calculating the position of an object appearing in the center of the data as a viewpoint and displaying the image data;
The notation conversion device is
Viewpoint determination means for determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of the plurality of users gazing at the display surface, which are acquired from the detection display device, are within the first display area. When,
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a determining means for determining as
notation conversion means for converting the first data written in the first display area into second data written in the second notation;
generating means for generating position information indicating the position of the second display area on the display surface;
an output means for outputting the second data and the position information to the detection display device;
the detection and display device receives the second data and the position information, and displays the second data in the second display area indicated by the position information ;
(a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in Chinese characters; The second notation is an expression in katakana, hiragana or romaji, (c) the first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, ( d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first The notation is an expression in a dialect, and the second notation is an expression in a standard language
A conversion display system characterized by: A control method used in a notation conversion device that converts first data notated in a first notation in a first display area on a display surface into notation in a second notation,
a viewpoint determination step of determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a decision step that decides as
a notation conversion step of converting the first data written in the first display area into second data written in the second notation;
a generating step of generating position information indicating the position of the second display area on the display surface;
an output step of outputting the second data and the position information to an external display device in order to display the second data in the second display area ;
(a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in Chinese characters; The second notation is an expression in katakana, hiragana or romaji, (c) the first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, ( d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first The notation is an expression in a dialect, and the second notation is an expression in a standard language
A control method characterized by: A computer-readable computer program recording a control computer program used in a notation conversion device that converts first data notated in a first display area on a display surface in a first notation into notation in a second notation a recording medium,
The notation conversion device, which is a computer,
a viewpoint determination step of determining whether or not a plurality of viewpoints among the respective viewpoints on the display surface of a plurality of users gazing at the display surface exist within the first display area;
Except for the first notation, only when there are multiple viewpoints within the first display area, the second notation is a notation that is compatible with multiple users who focus their eyes on the first display area. a decision step that decides as
a notation conversion step of converting the first data written in the first display area into second data written in the second notation;
a generating step of generating position information indicating the position of the second display area on the display surface;
an output step of outputting the second data and the position information to an external display device in order to display the second data in the second display area ;
(a) the first notation is an expression in a first language; the second notation is an expression in a second language; (b) the first notation is an expression in Chinese characters; The second notation is an expression in katakana, hiragana or romaji, (c) the first notation is an expression in katakana, hiragana or romaji, the second notation is an expression in kanji, ( d) the first notation is an expression in a standard language used as a national norm, the second notation is an expression in a dialect that has developed differently depending on the region, or (e) the first The notation is an expression in a dialect, and the second notation is an expression in a standard language
A recording medium recording a computer program characterized by :

Images