JPH08163122A - Remote conference method - Google Patents

Abstract

PURPOSE: To provide a remote conference system which is attained with use of a slow and inexpensive communication circuit and also needs a low cost for installation of terminals. CONSTITUTION: The terminal equipments 101 and 102 are connected together via a communication circuit 103. An input device 104 constructs a voice input means like a microphone, etc., a text like a keyboard, a mouse, etc., and a graphic information input means. The input data supplied from the device 104 are first outputted to an output device 106 of its own terminal through an output device control means 105 like an ordinary terminal equipment. The device 106 consists of a display and a speaker. At the same time, the input data that should be sent to the equipment 102 of the opposite party side are converted into a transmission data packet by a transmission data generation means 107 and sent to the opposite party terminal by a communication control means 108.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication method, and more particularly,
Allows users of multiple terminals connected by a communication line to carry out information processing jointly by performing data communication between terminals while sharing information while performing dialogue by voice. , The so-called remote conference method.

[0002]

2. Description of the Related Art In the past, a telephone has been most often used as a means for quick communication between remote places. The advantage of the telephone is that it can convey the reaction between the correspondents in real time, and uses voice, which is very easy for humans to understand. However, it is often very difficult to communicate when trying to discuss a complicated drawing only by telephone voice.

On the other hand, since electronic documents and drawings can be transmitted and received by electronic mail and facsimile, they are widely used as a means for sharing information at remote locations. However, these means are systems in which the sender and the receiver access information at different times, and cannot process information in real time.

Therefore, a system has been proposed in which, while transmitting and receiving voice, terminal devices connected by a communication line share document data to enable joint information processing in real time.

Two terminal devices connected by a communication line each have a telephone set, the same screen is displayed on the display devices of both terminals, and the cursor controlled by the own terminal and the other terminal are controlled on each display screen. A system configured to hold a conference in real time while referring to the screen by simultaneously displaying a cursor for
It is disclosed in Japanese Patent No. 084.

In the above system, since only the control information for displaying the same document as the voice on a plurality of terminals is transmitted / received, it is not necessary to transmit / receive a large amount of image information in real time unlike the so-called TV conference system. It is characterized by low equipment installation costs and that it can be implemented on existing communication equipment such as public telephone lines.

On the other hand, since the facial expressions and movements of the speaker are not always transmitted and received as images, especially when a teleconference is held among three or more people, the controller of the cursor displayed on the shared screen and the current speaker. Need to provide some means to identify

A means for specifying the controller of the cursor is disclosed in Japanese Patent Laid-Open No. 1-276892. The present invention provides means for identifying the controller of each cursor by allocating cursors with different marks such as shapes and colors to each conference participant.

Means for identifying the current speaker are disclosed in
It is disclosed in Japanese Patent Publication No. 64-49465. According to the present invention, a character string or graphic information for identifying each participant is displayed, and a speech mark is displayed near the character string or graphic information during a speech so that the present speaker can be recognized. Is provided.

[0010]

In the conventional invention proposed to identify the controller of the cursor, in the case where three or more people participate in the discussion at the same time, the cursor moves disorderly and the cursor mark moves. Even if there is a difference, there is a problem that it is difficult to naturally and quickly associate the cursor with the controller.

[0011] Further, when the actual process of debate is imagined, usually, the speaker is naturally recognized by paying attention to the movement of the speaker's facial expression, especially the movement centering around the mouth. Therefore, in the conventional invention in which the speaker is clearly indicated by the artificial mark display, there is a problem in that it becomes difficult to smoothly follow the flow of conversation when the speakers are frequently replaced. It was

[0012]

Therefore, in the present invention, a portrait (face, half-body or full-body image) of a conference participant is displayed on each terminal screen, and the portrait and the cursor controlled by each participant are displayed as follows. By displaying by connecting with a solid line, the controller of the cursor can be recognized at a glance.

Further, the user can freely obtain or abandon the shared screen control right. The cursor controlled by the user is displayed on the other user's terminal only when the user has the shared screen control right. It does not make the screen hard to see.

Further, in the present invention, a still image of a portrait of a conference participant is distributed to each terminal in advance, and the image is displayed on each terminal based on voice information and cursor movement information sent through a communication line. The above is transformed by computer graphics (CG) technology and displayed as a gesture or expression of the speaker.

[0015]

According to the present invention, the portrait of the conference participant and the cursor controlled by the participant are displayed by connecting with a solid line or the like, so that it is natural as if the participant points the screen with the pointing stick. The correspondence between each cursor and the participant is expressed in.

Further, since the gestures and facial expressions of the participants are pseudo-expressed by the CG processing, it is possible to identify the speaker in a form similar to that in a normal discussion. Since this CG processing is performed independently on each terminal, it is not necessary to send and receive image data during this processing.

Due to these actions, the naturalness of the actual discussion can be reproduced even in the teleconferencing system that does not constantly send and receive image data.

[0018]

1 is a block diagram of a communication terminal device as an embodiment of the present invention. In FIG. 1, two terminal devices 10
1, 102 are connected by a communication line 103. The input device 104 constitutes a voice input means such as a microphone and a text and graphic information input means such as a keyboard and a mouse. The input data input from the input means 104 is first output device control means 1 as in a normal terminal device.
It is output to the output device 106 of the own terminal through 05.
The output device 106 includes a display and a speaker. On the other hand, of the input data, the partner terminal 102
Need to be transmitted to the transmission data generating means 10
7, the data packet is converted into a data packet for transmission, and is transmitted to the partner terminal by the communication control means 108.

Similarly, a data packet is transmitted from the partner terminal 102. The received data analysis means 109 is
When the data packet is received, it is converted into a form that can be accepted by the output device control means 105 in the own terminal, and is output to the output device 106 through the output device control means 105.

In the device shown in FIG. 1, a remote conference is possible by reflecting the input from the own terminal on the output device of the other terminal. For simplification of the drawing, FIG. 1 shows a state in which two terminals are connected by a communication line, but the configuration of this figure can be immediately expanded to a form in which more terminals are connected.

FIG. 2 shows a typical display example on the display that is seen when performing a teleconference using the method and apparatus of the present invention. On the display 201,
Local window 202 that can be controlled only from its own terminal
The shared window 203 that can be controlled from other terminals is displayed. Prior to the execution of the remote conference, the same document is distributed to all users, and the same portion of this document is displayed in the shared window 203 of each user. Further, each user can edit the document in the shared window 203 and change the display portion, and the operation performed by a user on the document in the shared window 203 is immediately performed by all users. It is reflected in the shared window.

Next to the shared window 203, portraits 204, 205, 206 ... (Facial expressions, upper body image or whole body image) of each user are displayed. The image data of these portraits, like the document displayed in the sharing window,
It is distributed to all users in advance of the remote conference.

For the user who is currently talking, the speaker can be specified by increasing the brightness of the background of the portrait of this user. For example, FIG. 2 shows that user 1 is busy.

The cursor under the control of each user is normally displayed only on the display of the own terminal and not on the terminal screens of other users. For example, assume that FIG. 2 shows a display on the display of the user 2. In this case, the cursor 207 controlled by the user 2 is not displayed on the display of another user. Further, in this case, the user 2 cannot control the shared window. This is because it is possible to move the cursor casually during a meeting even if there is no intention, so reduce the number of cursors displayed at the same time to make the screen easier to see and to prevent the shared window from being operated. This is to prevent it.

Then, the user 2 shares the shared window 203.
If you want to explain to other users while pointing to a specific position in the document displayed in, declare it explicitly. Specifically, the shared window control start menu 208 is selected. When this menu is selected, the cursor 207 of the user 2 which has not been displayed on the other terminal until then is displayed on the terminal of the other user.

FIG. 3 shows the state. The position of the cursor (not shown) and the portrait 205 of the user 2 are displayed in a form connected by a solid arrow line 210. Since it seems that the pointer sticks to the document from the speaker, the correspondence between the speaker and the cursor is obvious. At the same time, the shared window control right is given to the user 2, and the shared window can be controlled by the user 2. That is, an operation performed on the document in the shared window by the user 2, for example, movement or deletion of the document area selected by the cursor or text input by the keyboard is reflected on the terminal screens of all the users. FIG. 3 shows a state in which the user 1 and the user 2 are in a conversation and both of them have the control right of the shared window.

It should be noted that the fact that the user who did not have a conversation suddenly starts operating the shared window gives a very abrupt impression to other users, and thus the selection of the shared window control menu can be accepted. It is natural to set certain conditions for. For example,
Only when the user is busy or within a predetermined time since the last utterance, the selection of the shared window control start menu by the user is accepted.

Further, in the present invention, in order to enhance the realism of the conference, the following two user interfaces are optionally provided.

In the first interface, as shown in FIG. 4, the mouth of the portrait of the user is moved and displayed in accordance with the voice uttered by each user. For example, an image of the lip portion 401 is cut out from the portrait 204, and the lip portion image is expanded or contracted vertically or horizontally depending on the vowel portion of the voice. This processing is executed by computer graphics (CG) processing on each terminal. Therefore, the voice transmitting side does not need to transmit the image data of the lips. In ordinary discussions, we follow the switch of speakers by unconsciously paying attention to the expression of the speaker, especially the movement of the mouth. The interface is intended to allow the flow of conversation to be grasped more naturally by artificially creating such an environment.

On the second interface, as shown in FIG. 5, the whole body image 501 of the conference participants by the CG image is displayed. For users who have shared window control,
An image in which the pointer 502 extends from the hand of this whole body image toward the position of the cursor controlled by this user.
The cursor position is represented. Then, in accordance with the cursor position, this whole-body image 501 of the user shows a movement such as changing the angle of the arm or changing the direction of the body. The image data of this movement is pseudo-generated as CG on each terminal based on the whole-body image of the user distributed in advance as a still image.

With this interface, on the terminal screen of each user, whether the meeting participants gather in front of a document displayed on a large screen such as an OHP display screen and discuss while pointing the screen with a pointing stick. An image like can be pseudo-generated.

FIG. 6 is a block diagram when the communication terminal device shown in FIG. 1 is configured by using a microprocessor. In FIG. 6, the microphone 601, as the input means,
A mouse 602 and a keyboard 603 are provided, and a display 604 and a speaker 605 are provided as output means. These devices are connected to the bus 611 via the interfaces 606, 607, 608, 609 and 610 for each input device. The CPU 612, which is connected to these devices through the bus 611, constitutes the transmission data generation means and the reception data analysis means of FIG. Then, the terminal device 600 transmits the full duplex line 615 via the communication interface 614.
Is connected to other similar terminal devices.

A processing method for realizing the user interface shown in FIGS. 2 to 5 by the terminal device having the configuration shown in FIG. 6 will be described with reference to a flowchart.

FIG. 7 is a flow chart of a control program executed by the CPU 612. When the terminal device is activated, the initial setting process 701 is first executed. Here, various settings are performed prior to the start of the conference, such as securing a communication channel with another terminal, opening a shared window and displaying distributed documents, and displaying portraits of conference participants. When this is finished, the program enters the interrupt wait state (step 702). If there is an end interrupt, step 70
In step 3, a series of termination processes such as releasing the communication channel, closing the shared window, and ending this program are performed.

When an interrupt other than the end is detected, a process according to the type of interrupt is performed. In this embodiment, a voice input interrupt (step 704) and a mouse interrupt (70
5) When the keyboard interrupt (706) and the data reception interrupt (707) are detected, the processing corresponding to each interrupt is executed asynchronously.

FIG. 8 is a flow chart showing the contents of the voice input interrupt routine. When the voice input from the microphone 601 of the terminal device of FIG. 6 is encoded into digital data by the decoder 606 and input to the bus 611, the CPU 612 detects this as an interrupt signal and activates this routine. In step 801, the CPU 612 reads the input voice data, generates a data packet to be transmitted to another terminal in step 802, and then executes step 8
At 03, a data packet is transmitted to another terminal. Finally, in step 804, the timer is reset and this routine is finished. Step 804 is provided to control the acceptance condition of the shared window control start command.

FIG. 9 shows the format of the data packet 900 generated in step 802. The CPU 612 adds an identifier 901 that uniquely identifies the source terminal to the header part. A data identifier 902 is added to the following area so that the receiving side can uniquely identify the type of data (in this case, the data is voice data). Then, the audio data 902 is concatenated to form one data packet.

FIG. 10 is a flow chart showing the contents of the mouse interrupt routine. In step 1001,
Interpret the input signal from the mouse and convert it to a command string. Here, the command character string is, for example, a combination of a mouse movement command and a new coordinate value of the mouse for a signal by moving the mouse, and for a signal of a mouse button click on the menu, This is the command corresponding to this menu. If this command is the shared window control start command, the process branches to step 1003.

Here, if the timer value is equal to or less than the set value, that is, if it is within a certain time since the user last uttered a voice, in step 1004, the shared window control right is given to the user and the process is terminated. Otherwise, the processing is immediately ended. If the command is the shared window control end command, the shared window control right of the user is revoked in step 1006.

For other commands, the following processing is performed. First, in step 1007, the command is executed and the execution result is output on the self-view screen. If the user does not have the shared window control right, this routine is finished.
If the user has the shared window control right, in step 1009, a solid line is drawn between the current position of the cursor controlled by the user and the portrait of the user.

Further, since it is necessary to reflect the execution result of the input command also on other terminals, step 101
When 0, the command character string is converted into a transmission data packet. This data packet has a format similar to that shown in FIG. 9, except that the data identifier is an identifier indicating mouse input and the data portion is the command character string generated in step 1001. And step 10
At 11, a data packet is transmitted to each terminal.

FIG. 11 shows the processing contents of the keyboard input interrupt routine. In step 1101, the input signal from the keyboard is interpreted and converted into a command character string. Specifically, a character string obtained by concatenating the text insertion command, the text insertion position, and the input text is generated as a command character string from the text input from the keyboard and the current cursor position. Step 1102
Then, the generated command is executed and the execution result is displayed on the terminal screen. Then, in step 1103, if the user does not have the shared window control right, this routine ends. If the user has the shared window control right, it is necessary to reflect the input to other terminals as well, so step 1104
To generate a data packet to send to another terminal,
In step 106, transmission processing is performed.

FIG. 12 is a flow chart showing the contents of the data reception interrupt routine. First, step 12
In 01, the transmission source of data is identified. Then step 12
At 02, the data identifier is read and the type of data is identified. Subsequent processing branches depending on the type of data. If the data is voice data, a voice signal, which is the data portion of the data packet, is output to the speaker through the decoder of FIG. 6 in step 1204. When the process of emphasizing the speaker is subsequently performed, the process proceeds to step 1205. Here, the background of the portrait of the speaker is emphasized, or as described with reference to FIG. 4, the image of the mouth of the portrait of the speaker is transformed by CG processing according to the type of voice, Processing such as pseudo-expressing a realistic sensation as if the speaker is actually speaking is performed.

If the data is keyboard input, the command character string which is the data part of the data packet is read in step 1207, and the command is executed in step 1208.

When the data is mouse input, the process branches as follows according to the contents of the command character string. If the command is a cursor movement command, the cursor is moved to a new position in step 1212. If the command is the shared window control start command issued by the user who is the data transmission source, a new cursor is drawn in step 1214. Continuing with steps 1212 and 1214, in order to specify the user who controls the cursor,
In step 1215, the cursor position and the portrait of the user who controlled the cursor are connected by a solid line and displayed.

Further, when the interface as described with reference to FIG. 7 is realized, the cursor control person emphasizing process is performed in step 1216. That is, here, the arm-raising angle of the portrait of the cursor control source user is calculated according to the cursor position, and the image of the portrait is updated by CG processing. If the command is the shared window control end command, the cursor under the control of the data source user is deleted in step 1218. If it is a command other than the above, it is executed in step 1219.

[0047]

According to the present invention, in realizing a natural dialogue environment close to an actual conference, it is not necessary to constantly send and receive the facial expressions of participants as moving images, so that the terminal installation cost is low. There is provided a remote conference system that can be realized by using a low-speed and inexpensive communication line.

[Brief description of drawings]

FIG. 1 is a block diagram of a remote conference terminal device of the present invention.

FIG. 2 is an explanatory diagram showing an example of a screen display during execution of a remote conference using the remote conference terminal device of the present invention.

FIG. 3 is an explanatory diagram of a second example of screen display during execution of a remote conference using the remote conference terminal device of the present invention.

FIG. 4 is an explanatory diagram of a portrait of a user.

FIG. 5 is an explanatory diagram of conference participants.

FIG. 6 is a block diagram when the communication terminal device is configured using a microprocessor.

FIG. 7 is a flowchart of a control program executed by a CPU 612.

FIG. 8 is a flowchart showing the contents of a voice input interrupt routine.

FIG. 9 is a data packet 9 generated in step 802.
Explanatory drawing which shows the format of 00.

FIG. 10 is a flowchart showing the contents of a mouse interrupt routine.

FIG. 11 is an explanatory diagram of processing contents of a keyboard input interrupt routine.

FIG. 12 is a flowchart showing the contents of a data reception interrupt routine.

[Explanation of symbols]

 104 ... Input device, 106 ... Output device.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical indication H04N 7/15

Claims (6)

[Claims] 1. A plurality of terminal devices are connected by a communication line, and each of the terminal devices has a display screen, pointing means for indicating a position on the display screen, data and command input means, and There is a shared window that can be controlled from other terminals on the screen, and the user of each terminal device can obtain and abandon the operation right of the shared window by entering a command. While acquiring the operation right of the above, this user executes the operation procedure of the shared window performed by the user using the pointing means or the input means on his / her own terminal, and the operation procedure of the shared window is used as packet data in another terminal. On other terminals, the operation procedure of the shared window is restored from the received packet data and executed. By reproducing the state of the shared window of the terminal that transmits the packet data in the shared window of another terminal, communication between the users of the terminals can be achieved through information on the shared window. And the method of remote conference. 2. The method according to claim 1, wherein while the user of the terminal device obtains the operation right of the shared window,
A cursor is displayed not only on the own terminal but also on the display screens of other terminals at the position on the screen instructed by the pointing means, and when the shared window operation right is abandoned, the cursor is displayed on the display screen of the own terminal. The method of teleconferencing that only appears above. 3. The remote conference method according to claim 1, wherein each of the terminal devices has voice input / output means, and the packet data and the voice are transmitted / received in parallel via the communication line. 4. The method according to claim 3, wherein each of the terminal devices has a timer, the value of the timer is reset every time a voice is transmitted from the terminal, and the value of the timer becomes a predetermined value or more. There is a method of remote conferencing in which the shared window control right cannot be acquired. 5. The portrait according to claim 2, wherein the portraits of the users of the respective terminal devices are arranged around the shared window to form a conference block diagram, and a certain user has acquired the shared window operation right. In this case, by drawing a display on the display screen of each terminal device, it is clear that the cursor position controlled by this user and the portrait of this user are geometrically connected, A teleconferencing method that specifies the correspondence between the cursor and its controller. 6. The method according to claim 3, wherein a portrait of the user of each of the communication terminals is arranged around the shared window to form a block diagram of the conference, and a user is transmitting voice, or If you are controlling the shared window,
According to this user's input, all or part of the portrait of this user is transformed and displayed by computer graphics processing on each terminal side, and is displayed to other users. A method of teleconferencing that artificially expresses that a person is performing an input operation.