JP2010165152A - Terminal device, screen transmission system, terminal program, and screen transmission method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce data transmitted or received when a screen to be displayed on a terminal device is updated, and to efficiently perform processing required for compensating the reduced data. <P>SOLUTION: An update data transmission unit 122 in an apparatus 110 transmits update data which indicates the display content in an HMI screen area updated by a server screen update unit 121. Everytime the update data is received by an update data reception unit 136, a terminal screen update unit 137 in a remote terminal 130 selects one of rendering layer objects 135 in an arbitrary sequence. The terminal screen update unit 137 then displays the display content, indicated by the update data, within an area in the selected rendering layer object 135 which corresponds to an area whose display content is indicated by the update data. Further, the terminal screen update unit 137 updates the terminal screen by overlapping the plurality of rendering layer objects 135 in such a manner that the selected rendering layer object 135 comes the top front face, and outputs the updated terminal screen on a display device. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a terminal device, a screen transmission system, a terminal program, and a screen transmission method. The present invention particularly relates to a device having an HMI (Human / Machine / Interface) screen, a remote terminal for monitoring and operating the HMI screen in real time from a remote location, and a system between the two. . In particular, the present invention relates to a change area screen transmission method and display method to a remote terminal.

  Devices used in FA (Factory Automation) and PA (Process Automation) generally have many setting items, and it is rare that end users can use the functions of the product immediately after installing the product. For this reason, as part of support for manufacturers, lectures on usage are often given to end users in remote locations such as overseas. Also, as a maintenance service, there are many cases in which the status of an end user's device is diagnosed or a failure occurrence status is confirmed from a remote terminal. For this reason, there is a demand for monitoring and operating the HMI screen of the device from a remote terminal.

  As a method for monitoring and operating the HMI screen of a device from a remote terminal, it has been necessary to install a dedicated application for remote monitoring and operation on a PC (personal computer), which has been a remote terminal. As a result, the development cost of this dedicated application is high, and it takes time and effort for the user to install the dedicated application for free or for a fee. In addition, if the manufacturer wants to change the transmission method of the HMI screen between the device and the remote terminal due to the manufacturer's reasons, for example, a new method for transmission data reduction or security improvement, the user is forced to upgrade all dedicated applications. It was necessary and inconvenient.

  However, at present, Web browsers are widely used as general-purpose display tools that operate on various platforms. The display processing on the Web browser is realized only by a JavaScript (registered trademark) or HTML (HyperText, Markup, Language) document stored in a device on which the Web server operates. There is also an advantage that the transmission method of the HMI screen can be freely changed within a range defined by (HyperText / Transfer / Protocol).

  In addition, since the amount of image data is enormous, a method has been devised in which the size is reduced, the time required for transmission and reception is reduced, and traffic is not affected. In particular, when performing remote diagnosis by HTTP using a Web browser, “reducing transmission line traffic” by reducing the amount of image data transmitted from a device on which the Web server operates, The display performance on the Web browser is improved by two measures of “efficient use of the transmission path” by optimizing the request timing.

  Regarding “reduction of transmission path traffic”, there is a method disclosed in Patent Document 1, for example, as a method of reducing transmission image data and transmitting HMI screen information to a remote terminal. In Patent Document 1, the first electronic device transmits a full screen frame to the second electronic device, and at least one rectangular data including a changed rectangle indicating an area where the change has occurred in the full screen frame is later transmitted to the second electronic device. The amount of screen frame data to be transmitted is effectively reduced.

  As a method of combining and displaying a plurality of images on a Web browser, for example, there is a method disclosed in Patent Document 2. Japanese Patent Laid-Open No. 2004-228561 presents information combining means for displaying a plurality of information updated at different periods on a Web browser in one display area.

JP 2006-318451 A JP 2001-125628 A

  HTTP used in WWW (World Wide Web) operates by a request-response format mechanism in which a Web browser transmits a request and a Web server returns a response. In this case, it does not make much sense if the time interval of requests transmitted from the Web browser does not correspond to the amount of screen frame data to be transmitted. That is, with the technique disclosed in Patent Document 1, even if the data amount is reduced, only the traffic on the transmission path is reduced and the influence on other communications is reduced, and the performance of the target system itself is improved. is not. Further, in order to display the same content as the HMI screen of the device on which the Web server operates with the Web browser of the remote terminal, it is necessary to continuously superimpose and display the transmitted region image data of the changed portion. is there.

  Japanese Patent Laid-Open No. 2004-259561 discloses a technique for processing information transmitted from a Web server as an HTML or XML (extensible Markup Language) document using ActiveX (registered trademark) or the like in the Web browser. In this method, image data to be displayed is created and displayed in one display area, and the image data itself transmitted from the Web server is continuously superimposed and displayed as will be described later as an embodiment of the present invention. Not suitable as a method. In addition, there is no disclosure of a method of displaying the image data transmitted to the Web browser differently each time and displaying the image data semi-permanently in front of the image data requested in the past and transmitted as a response. .

  An object of the present invention is, for example, to reduce data transmitted or received when updating a screen to be displayed on a terminal device, and to improve the efficiency of processing necessary to supplement the reduced data. .

A terminal device according to one aspect of the present invention includes:
In a terminal device that outputs a terminal screen configured to superimpose a plurality of layers capable of displaying an image and hide an area where an image displayed on a front layer overlaps with a back layer,
An update data receiving unit for receiving update data indicating display contents after update of an area to be updated in the terminal screen;
Each time update data is received by the update data receiving unit, one layer is selected from the plurality of layers in an arbitrary order, and the display content indicated by the update data is an area in the selected layer, The updated data is displayed in an area corresponding to the area indicating the display content, and the terminal screen is updated by superimposing the plurality of layers so that the selected layer is in front of the previously selected layer, and the updated terminal And a terminal screen update unit that outputs the screen to a display device.

  According to one aspect of the present invention, in a terminal device that outputs a terminal screen configured to superimpose a plurality of layers and hide an area in which an image displayed on the front layer overlaps the back layer, the terminal The screen update unit selects one layer from the plurality of layers in any order and receives the update data each time update data indicating the display content after updating the region to be updated in the terminal screen is received. The display content indicated by the data is displayed in an area in the selected layer corresponding to the area in which the update data indicates the display content, and the selected layer is placed in front of the previously selected layer. By updating the terminal screen by superimposing a plurality of layers, it is necessary to reduce data transmitted or received when updating the terminal screen and to supplement the reduced data. It is possible to improve the efficiency of the become processing (processing for combining the post-updated data before updating data).

1 is a block diagram showing a configuration of a screen transmission system in Embodiment 1. FIG. 2 is a diagram illustrating an example of a hardware configuration of a device and a remote terminal according to Embodiment 1. FIG. 6 is a diagram showing a method for calculating an update area from a drawing area history in Embodiment 1. FIG. 5 is a diagram showing a method for continuously superimposing and displaying update areas on the Web browser in Embodiment 1. FIG. FIG. 6 is a flowchart (one request) when the interval between screen update requests in the first embodiment is constant. FIG. 10 is a flowchart (1 request) when the interval between screen update requests in the first embodiment is variable. FIG. 10 is a flowchart (2 requests) when the interval between screen update requests in the first embodiment is constant. FIG. 10 is a flowchart (2 requests) when the interval between screen update requests in the first embodiment is variable. FIG. 6 is a flowchart (1 request) showing a method for determining a screen update request interval time in the first embodiment. FIG. 5 is a flowchart (2 requests) illustrating a method for determining a screen update request interval time in the first embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a screen transmission system 100 according to the present embodiment.

  In FIG. 1, a screen transmission system 100 includes a device 110 having an HMI screen to be remotely monitored, and a remote terminal 130 (an example of a terminal device) that monitors the HMI screen of the device 110.

  The device 110 is an example of a server device that updates a server screen (HMI screen) that can display an image, and includes a server screen update unit 121 and an update data transmission unit 122.

  The server screen update unit 121 of the device 110 includes a main control unit 114 of the device 110 main body, an input command processing unit 113 that processes input information from an input device such as a key attached to the device 110 or a touch panel, and waits for the command. The input command event queue 111 held as a matrix, the HMI screen drawing unit 115 for displaying the HMI of the device 110, and the HMI screen data 116 stored in the VRAM (Video, Random, Access, Memory) included in the device 110. And have. Further, the server screen update unit 121 of the device 110 has drawing area history data 112 that records the area where the HMI screen drawing unit 115 has drawn.

  The server screen update unit 121 of the device 110 updates the HMI screen using the above-described units.

  The update data transmission unit 122 of the device 110 has a screen capture processing unit 118 for capturing an HMI screen. In addition, the update data transmission unit 122 of the device 110 includes an HTTP server function unit 119 as a communication function, and a CGI (Common Gateway Interface) program 117 that processes a request from an HTTP client that is the remote terminal 130. Is working. Specifically, the HTTP server function unit 119 and the CGI program 117 are implemented in a Web server that operates on the device 110. The update data transmission unit 122 of the device 110 is an HTML document transmitted to the Web browser of the remote terminal 130 at a timing when accessed from the client or a JavaScript for remote display processing, and is a storage device included in the device 110. The HTML / JavaScript document 120 is stored in the file.

  The update data transmission unit 122 of the device 110 generates update data using the above-described units, and transmits the generated update data to the remote terminal 130. Here, the update data is data indicating the display content of the area updated by the server screen update unit 121 in the HMI screen. For example, coordinate data indicating the position of the area in the HMI screen, It includes a pixel data group.

  On the other hand, the remote terminal 130 outputs a terminal screen (browser screen) configured to superimpose a plurality of layers capable of displaying an image and hide an area where the image displayed on the front layer overlaps the back layer. The update data receiving unit 136 and the terminal screen update unit 137 are provided.

  The update data receiving unit 136 of the remote terminal 130 is equipped with an HTTP client function unit 131. Specifically, the HTTP client function unit 131 is implemented in a Web browser that operates on the remote terminal 130.

  The update data reception unit 136 of the remote terminal 130 receives the update data transmitted from the device 110 by the update data transmission unit 122 using the HTTP client function unit 131 described above.

  The terminal screen update unit 137 of the remote terminal 130 includes an HTML cache 134 that temporarily stores the HTML / Java Script document 120 acquired from the Web server, and an HTML / Java Script analysis unit 132 that analyzes and processes the HTML / Java Script document 120. And a browser display processing unit 133 that displays image objects and the like on the browser using a plurality of drawing layer objects 135 (an example of a plurality of layers). The HTML / Java Script analysis unit 132 and the browser display processing unit 133 are also implemented in a Web browser that operates on the remote terminal 130.

  The terminal screen update unit 137 of the remote terminal 130 first selects one drawing layer object 135 from a plurality of drawing layer objects 135 in an arbitrary order each time update data is received by the update data receiving unit 136. Next, the terminal screen update unit 137 identifies an area in the selected drawing layer object 135 corresponding to the area where the update data received by the update data reception unit 136 indicates the display content. For example, the terminal screen update unit 137 refers to the coordinate data included in the update data, and specifies an area at the same position in the selected drawing layer object 135. Next, the terminal screen update unit 137 displays the display content indicated by the update data received by the update data reception unit 136 in an area in the specified drawing layer object 135. For example, the terminal screen update unit 137 displays all pixel data included in the update data in the specified area. Next, the terminal screen update unit 137 superimposes the plurality of drawing layer objects 135 so that the selected drawing layer object 135 comes to the front of the previously selected drawing layer object 135 and updates the browser screen. At this time, the terminal screen update unit 137 may simply move the selected drawing layer object 135 to the foreground, as will be described later. Finally, the terminal screen update unit 137 outputs the updated browser screen to the display device.

  An HTTP request 141 is transmitted from the HTTP client function unit 131 of the remote terminal 130 to the HTTP server function unit 119 of the device 110, and an HTTP response 142 is returned as a response thereto.

  The update data receiving unit 136 of the remote terminal 130 transmits request data for requesting transmission of update data to the device 110 as one of the HTTP requests 141. The update data transmission unit 122 of the device 110 receives the request data transmitted from the remote terminal 130 by the update data reception unit 136, and transmits the update data to the remote terminal 130 as an HTTP response 142 in response to the request data. . The update data transmission unit 122 transmits the update data to the remote terminal 130 every time request data is received.

  In the present embodiment, the update data transmission unit 122 of the device 110 transmits full screen data as update data at least once out of a plurality of times of transmission of update data. Therefore, the update data receiving unit 136 of the remote terminal 130 receives full screen data as update data at least once out of receiving update data from the device 110 a plurality of times. Here, the full screen data is data indicating the display contents of the entire HMI screen, and includes, for example, all pixel data groups in the HMI screen.

  It is desirable that the update data receiving unit 136 of the remote terminal 130 receives the full screen data as the update data at least once out of receiving the update data the number of times equal to the number of the drawing layer objects 135. That is, it is desirable that the update data receiving unit 136 receives full screen data as update data at least once before the selection of the drawing layer object 135 by the terminal screen update unit 137 is completed. For example, the update data receiving unit 136 may receive the full screen data first, and then receive the full screen data again when the selection of the drawing layer object 135 by the terminal screen update unit 137 is completed.

Further, in the present embodiment, the update data transmission unit 122 of the device 110 is further updated based on the size of the area updated in the HMI screen every time the HMI screen is updated by the server screen update unit 121. The time required for transmission of the update data indicating the display contents of the area is calculated as the required transmission time. As will be described later, the update data transmission unit 122 may instruct the remote terminal 130 when to transmit the next request data after receiving the update data, based on the calculated required transmission time. In this case, when the update data transmission unit 122 transmits the HTTP response 142 including the update data, the update data transmission unit 122 adds data instructing the remote terminal 130 to the time when the next request data should be transmitted to the HTTP response 142.
Alternatively, only the update data size may be transmitted from the device 110, and the next update data request timing may be calculated from the update data size on the remote terminal 130 side.

  In the present embodiment, the update data transmission unit 122 of the device 110 is updated to the remote terminal 130 by the server screen update unit 121 in the HMI screen instead of the update data based on a predetermined condition described later. Sampling data obtained by performing predetermined sampling with respect to the display content of the area may be transmitted.

  Note that this embodiment is also applicable when the update data receiving unit 136 of the remote terminal 130 receives update data from other than the device 110.

  FIG. 2 is a diagram illustrating an example of a hardware configuration of the remote terminal 130.

  In FIG. 2, each of the remote terminals 130 is a computer, and includes an LCD 901 (Liquid / Crystal / Display), a keyboard 902 (K / B), a mouse 903, an FDD 904 (Flexible / Disk / Drive), and a CDD 905 (Compact / Disk / Drive). ) And a hardware device such as a printer 906. These hardware devices are connected by cables and signal lines. Instead of the LCD 901, a CRT (Cathode / Ray / Tube) or other display device may be used. Instead of the mouse 903, a touch panel, a touch pad, a trackball, a pen tablet, or other pointing devices may be used.

  The remote terminal 130 includes a CPU 911 (Central / Processing / Unit) that executes programs. The CPU 911 is an example of a processing device. The CPU 911 includes a ROM 913 (Read / Only / Memory), a RAM 914 (Random / Access / Memory), a communication board 915, an LCD 901, a keyboard 902, a mouse 903, an FDD 904, a CDD 905, a printer 906, and an HDD 920 (Hard / Disk) via a bus 912. Connected with Drive) to control these hardware devices. Instead of the HDD 920, a flash memory, an optical disk device, a memory card reader / writer, or other storage medium may be used.

  The RAM 914 is an example of a volatile memory. The ROM 913, the FDD 904, the CDD 905, and the HDD 920 are examples of nonvolatile memories. These are examples of the storage device. The communication board 915, the keyboard 902, the mouse 903, the FDD 904, and the CDD 905 are examples of input devices. The communication board 915, the LCD 901, and the printer 906 are examples of output devices.

  The communication board 915 is connected to a LAN (Local / Area / Network) or the like. The communication board 915 is not limited to a LAN, but is an IP-VPN (Internet, Protocol, Private, Network), a wide area LAN, an ATM (Asynchronous / Transfer / Mode) network, or a WAN (Wide / Area / Network) It does not matter if it is connected to. LAN, WAN, and the Internet are examples of networks.

  The HDD 920 stores an operating system 921 (OS), a window system 922, a program group 923, and a file group 924. The programs in the program group 923 are executed by the CPU 911, the operating system 921, and the window system 922. The program group 923 includes programs that execute the functions described as “˜units” in the description of the present embodiment. The program is read and executed by the CPU 911. The file group 924 includes data, information, and signal values described as “˜data”, “˜information”, “˜ID (identifier)”, “˜flag”, and “˜result” in the description of this embodiment. And variable values and parameters are included as items of “˜file”, “˜database”, and “˜table”. The “˜file”, “˜database”, and “˜table” are stored in a storage medium such as the RAM 914 or the HDD 920. Data, information, signal values, variable values, and parameters stored in a storage medium such as the RAM 914 and the HDD 920 are read out to the main memory and the cache memory by the CPU 911 via a read / write circuit, and extracted, searched, referenced, compared, and calculated. It is used for processing (operation) of the CPU 911 such as calculation, control, output, printing and display. During the processing of the CPU 911 such as extraction, search, reference, comparison, calculation, calculation, control, output, printing, and display, data, information, signal values, variable values, and parameters are temporarily stored in the main memory, cache memory, and buffer memory. Remembered.

  The arrows in the block diagrams and flowcharts used in the description of this embodiment mainly indicate input / output of data and signals. Data and signals are recorded in memory such as RAM 914, FDD904 flexible disk (FD), CDD905 compact disk (CD), HDD920 magnetic disk, optical disk, DVD (Digital Versatile Disc), or other recording media Is done. Data and signals are transmitted by a bus 912, a signal line, a cable, or other transmission media.

  In the description of the present embodiment, what is described as “to part” may be “to circuit”, “to device”, “to device”, and “to step”, “to process”, “to”. ~ Procedure "," ~ process ". That is, what is described as “˜unit” may be realized by firmware stored in the ROM 913. Alternatively, what is described as “˜unit” may be realized only by software, or only by hardware such as an element, a device, a board, and wiring. Alternatively, what is described as “to part” may be realized by a combination of software and hardware, or a combination of software, hardware and firmware. Firmware and software are stored as programs in a recording medium such as a flexible disk, a compact disk, a magnetic disk, an optical disk, and a DVD. The program is read by the CPU 911 and executed by the CPU 911. That is, the program causes the computer to function as “to part” described in the description of the present embodiment. Or a program makes a computer perform the procedure and method of "-part" described by description of this Embodiment.

  FIG. 3 is a diagram illustrating a method of transmitting the HMI screen information to the remote terminal 130 by reducing the transmission image data in order to reduce the transmission path traffic.

  In FIG. 3, a capture process 220 (executed by the screen capture processing unit 118) and a rendering process 230 (HMI screen), which are processes on the side of the device 110 having the Web browser 210 on the remote terminal 130 and the HMI screen 240. And the state of the HMI screen 240 rendered thereby is shown in time series. Screen capture requests A211, A212, A213, and A214 (all of which are HTTP requests 141) are transmitted from the Web browser 210 to the capture processing 220. On the other hand, in the drawing process 230, drawing commands C231, C232, C233, and C234 are executed on the HMI screen 240 asynchronously. In the example of FIG. 3, C231 is rectangular drawing, C232 is circular drawing, C233 is triangular drawing, and C234 is rectangular deletion. The initial state of the HMI screen 240 is D241, which has been changed to screens D242, D243, D244, and D245 by drawing commands C231, C232, C233, and C234, respectively.

  In FIG. 3, A211, A212, A213, and A214 are examples of request data. B221 is an example of display contents of the entire HMI screen 240 indicated by the full screen data. B222, B223, and B224 are examples of display contents of an area updated by the server screen update unit 121 in the HMI screen 240 indicated by the update data.

  A flow from a screen capture request to image data transmission will be described below. Here, a case where the screen capture request A213 is transmitted will be described.

  First, the screen capture request A 213 is transmitted from the Web browser 210 to the capture processing 220 of the device 110 as an HTTP request 141. The capture process 220 acquires the drawing area drawn on the HMI screen 240 by the drawing process 230 over a period from the reception of the previous screen capture process request A212 to the reception of the current request A213. In this example, the HMI screen 240 is in the state of D243 by drawing the circle of the drawing command C232 during this period, and the HMI screen 240 is in the state of D244 by drawing the triangle of the drawing command C233 subsequently.

  Next, the capture processing 220 determines the minimum rectangular area B223 including the drawing area drawn by these two drawing commands C232 and C233 by coordinate calculation, and captures the HMI screen 240 for this area. The captured image data is transmitted to the Web browser 210 as an HTTP response 142. At this time, areas drawn by two drawing commands of C232 and C233 may be separately captured and transmitted to the Web browser 210. In this case, separately from the image capture request, a screen change area acquisition request is transmitted from the Web browser 210 in advance, and the capture processing 220 is prompted to capture a plurality of areas as a response thereto.

  In addition to the normal method of capturing all pixels (pixels) on the HMI screen 240, the capture processing 220 is performed when the number of pixels in the target region is equal to or greater than a preset threshold (an example of a predetermined condition), or a device In consideration of the communication status between the network 110 and the remote terminal 130 and the display performance of the Web browser 210 that has made the screen capture request, when these performances (for example, communication speed and display speed) are below a preset threshold (predetermined conditions) In an example, the HMI screen 240 captures by thinning out at a rate of 1 pixel in an area of vertical M pixels (M is an integer of 2 or more) and horizontal N pixels (N is an integer of 2 or more) (for example, There is also a method that captures only 2 pixels diagonally per 2 × 2 pixels.

  In order to display the correct position on the Web browser 210 on the remote terminal 130, the display position (XY coordinates) of the transmitted image data in the HMI screen 240 also needs to be transmitted to the Web browser 210. In addition to the screen capture request, a screen change area acquisition request may be transmitted from the Web browser 210 in advance and notified to the Web browser 210 as a response to this (an example of coordinate data). Coordinate information (another example of coordinate data) may be included as cookie information or the like in the HTTP header at the time of data transmission and notified at the same time as image data transmission. Here, the former is called a two-request method, and the latter is called a one-request method.

  In addition, when drawing is not performed during the period from the previous screen capture request reception to the current screen capture request reception, the HMI screen 240 is not updated, so an empty HTTP response 142 is returned to the Web browser 210.

  In FIG. 3, the full screen capture request A211 is periodically transmitted from the web browser 210 to the capture processing 220 of the device 110 as an HTTP request 141. In this case, the full screen capture of the HMI screen 240 at the time of receiving the full screen capture request is performed regardless of the drawing process from the previous capture request, and the resulting image data B 221 is transmitted to the Web browser 210.

  Next, a method for displaying the transmitted area image data on the remote terminal 130 will be described.

  FIG. 4 is a diagram showing a method for displaying a plurality of images superimposed on the Web browser 210.

  The Web browser 210 can display a finite number (drawing four in FIG. 4 for illustration) of drawing layer objects 135 in a superimposed manner. In FIG. 4, E201 which is a drawing layer object 135 is displayed on the full screen on the back side, white rectangles are displayed with E202 on the front side, white circles and triangles are displayed with E203, and rectangles are erased with E204 (the same color as the background is drawn). The drawing layer object 135 is superimposed and displayed in the order of As a result, at the time of FIG. 4, it appears that a white circle and a triangle are drawn on the colored background on the Web browser 210. This is the same state as D245 on the HMI screen 240 in FIG.

  Hereinafter, a method for superimposing and displaying image data in the Web browser 210 will be described.

  Consider a case where image data B221, B222, B223, and B224 are sequentially received from the capture process 220 in response to screen capture requests A211, A212, A213, and A214 from the Web browser 210 as shown in FIG.

  First, when reception of the first image data B221 from the capture processing 220 is completed, this image data is read into E201 which is a drawing layer object 135 displayed on the backmost surface of the Web browser 210 and moved to the frontmost surface. At the same time, it is displayed (F211). Similarly, when the reception of the next image data B222 is completed, the drawing layer object 135 is read into E202, moved to the foreground, and displayed (F212). In this way, the transmitted image data is assigned to the drawing layer object 135 on the backmost surface every time, and when the reception of the image data is completed, the image data received is moved to the foreground and displayed. The past image data remains visible in the area where the data is always displayed in the foreground and is not included in the received image data (not updated on the HMI screen 240 of the remote device 110). Therefore, the latest HMI screen 240 can always be remotely displayed to the user viewing the Web browser 210 (browser screen).

  In the example of FIG. 4, since there are only four drawing layer objects 135, it is necessary to reuse E201 which is the drawing layer object 135 on the back after the drawing layer object 135 of E204 is moved and displayed on the foreground. Occurs. At this time, similarly to the movement / display of E204, when only the area changed on the HMI screen 240 side of the device 110 is received as image data and assigned to E201 which is the drawing layer object 135, the movement / display is performed ( F215), the background information of the entire screen that has been displayed in the past disappears, only the update area for the last four times is displayed, and the rest is displayed as a blank superimposed display.

  For this reason, as described above, it is necessary to receive the image data of the entire screen every time the number of drawing layer objects is reached. As this method, as shown in the above example, a method of always requesting a full screen update request from the Web browser 210 for each number of drawing layer objects, or an image returned in response to each screen capture request The size of data is confirmed on the Web browser 210 side, and after the last full screen data is sent, a full screen capture request is sent for each number of drawing layer objects.

  Next, the method for determining the screen update request interval time for “efficient use of the transmission path” and its effect will be described separately for each of the 1-request method and the 2-request method.

  FIG. 5 shows screen update request transmission from the Web browser 310 (A311, A312, A313, A314, A315) and image data transmission from the Web server 320 (B321, B322, B323, B324, B325) in one request method. It is the figure which represented the example in time series. In this figure, requests from the Web browser 310 are transmitted at equal time intervals.

  In FIG. 5, first, the image (B321) of the entire screen is transmitted in response to the request of A311. Since the subsequent requests of A312, A313, A314, and A315 are only part of the update in the screen, the transmission data It represents that the amount (B322, B323, B324, B325) is decreasing. In this example, between the responses of B322, B323, B324, and B325, the amount of transmission data on the transmission path between the Web browser 310 and the Web server 320 is small and the efficiency is poor. This is because a screen update request from the Web browser 310 is periodically transmitted.

  FIG. 6 is a diagram showing an example in which the screen update request timing from the Web browser 410 is dynamically determined and changed under the same conditions as FIG.

  The size of the image data to be transmitted is known by the Web server 420 in the one-dot broken line portions of the responses B421, B422, B423, and B424 from the Web server 420. For this reason, the time interval until the next request transmission by the Web browser 410 is calculated from this data size and specified to the Web browser 410 side, thereby shortening the screen update interval in the Web browser 410 and using the transmission path. Efficiency can also be improved.

  FIG. 7 shows screen update request transmission (A511, A512, A513, A514, A515) from the web browser 510 and image data transmission (B521, B522, B523, B524, B525) from the web server 520 in the two-request method. It is the figure which represented the example in time series. In this figure, requests from the Web browser 510 are transmitted at equal time intervals.

  In FIG. 7, first, the position and size of the area updated in the Web server 520 are calculated by the screen update area request (HTTP request 141) of A511, and the coordinates and size of the area are set to the response C526 (HTTP The response 142) is returned to the web browser 510. The web browser 510 transmits an image data request D527 (HTTP request 141) based on this. The Web server 520 captures the range of the area designated by D527 (HTTP request 141) and transmits the image data by response B521 (HTTP response 142). The same applies to A512, A513, A514, and A515. In the case of this example as well, the load on the transmission path between the Web browser 510 and the Web server 520 is low and the efficiency is low as in the example of FIG.

  FIG. 8 is a diagram illustrating an example in which the screen update request timing from the Web browser 610 is dynamically determined and changed under the same conditions as in FIG.

  The response C626 from the Web server 620 to the first screen update area request in the two-request method, and the subsequent responses C627, C628, C629, and C630 (dotted line portion) are B621, B622, B623, B624, and B625 in the Web server 620. The size of the image data to be transmitted is known. For this reason, the time interval until the next request transmission by the web browser 610 is calculated from this data size and specified to the web browser 610 side, thereby shortening the screen update interval in the web browser 610 and using the transmission path. Efficiency can also be improved.

  Hereinafter, the operation of the screen transmission system 100 illustrated in FIG. 1 will be described in order.

  First, the user of the remote terminal 130 accesses the address of the HTTP server function unit 119 operating on the device 110 that is a remote diagnosis target from the Web browser. Then, the HTML / Java Script document 120 in the server is stored in the HTML cache 134 of the remote terminal 130. Next, the HTML / Java Script document 120 is analyzed by the HTML / Java Script analysis unit 132 and the script is executed.

  In JavaScript, Ajax (Asynchronous, JavaScript, and XML) technology is used to generate a screen update request to the server at a certain time interval T asynchronously with the processing on the server side. The value of T at this time can be dynamically changed according to the amount of data transmitted from the server to the remote terminal 130, as shown in FIG. 9 or FIG. In addition, the screen update request can specify as a request parameter whether to update the entire screen or only the drawing area.

  On the other hand, in the device 110 (especially when monitoring a drive system controlled by the device 110), the HMI screen drawing unit 115 always draws the HMI screen data 116 in response to a command from the main control unit 114. Yes. The drawing command and information about the drawing target area are sequentially stored in the drawing area history data 112.

  Hereinafter, the case of the 1 request method and the case of the 2 request method will be described separately.

  In the case of the one request method, the HTTP server function unit 119 of the device 110 that has received a screen data request from the Web browser of the remote terminal 130 calls the CGI program 117 that is a request handler. The corresponding CGI program 117 determines from the CGI call parameter whether the screen data request is for the entire screen or only for the drawing area. If it is a full screen, the screen capture processing unit 118 is requested to capture.

  When only the drawing area is targeted, the screen capture processing unit 118 refers to the drawing area history data 112 held by the HMI screen drawing unit 115 and includes the minimum drawing area including the accumulated drawing area from the previous request. The coordinates of are calculated. Next, the calculated area portion is captured.

  Note that the amount of image data to be transmitted later can be known from the rectangular information of the capture area, and the next capture timing transmitted from the Web browser can be calculated from the data amount.

  FIG. 9 is a diagram showing a method for determining the screen data request interval time in the case of the one request method for “efficient use of the transmission path”.

  As described above, the Web browser 710 operates on the remote terminal 130, and the Web server 720 operates on the device 110. A711 and A712 are screen data requests, B721 and B725 are screen update area determination processing, C722 and C726 are screen capture processing, D723 and D727 are time interval designation responses until the next request, and E724 and E728 are screen data transmission processing.

  The transmission delay time between the Web browser 710 and the Web server 720 in the example of this figure is α. Further, the time required for the screen update area determination processing B721 and B725 is denoted by Δa. Further, the time required for the screen capture processing C722 and C726 is assumed to be Δb1 and Δb2, respectively. Further, let d be the time taken for screen data transmission E724.

  The interval between the first screen data request A711 and the second screen data request A712 in FIG. 9 is specified so that the next screen data transmission E728 is started immediately after the completion of the screen data transmission E724. The efficiency of the transmission line is maximized. Hereinafter, the interval between the response D723 reception and the request A712 transmission in the Web browser 710 at this time is formulated.

  The time interval between screen data requests is designated to the Web browser 710 by D723 which is a response from the Web server 720. Since the value specified at this time needs to be the time from the timing at which the response D723 is interpreted by the Web browser 710, the time in the range of F729 in FIG.

From the above conditions, the following equation (1) is established.
Formula (1): F729 + A712 + B725 + C726 = E724-D723

When a variable is assigned after setting the time of F729 as x, the following equation (2) is obtained.
Formula (2): x + α + Δa + Δb2 = d−α

Therefore, the following expression (3) is obtained.
Formula (3): x = d−2α−Δa−Δb2

Here, since the value of Δb2 is unpredictable at the time of B721, the Web server 720 that processes this calculation formula considers Δb2 = Δb1, and furthermore, the screen update area determination processing (B721 and B725) can be ignored. Therefore, when Δa = 0, the following equation (4) is obtained.
Formula (4): x = d−2α−Δb1

  Further, the value of the screen data transmission time d (an example of the required transmission time) is proportional to the amount of transmission data, and the amount of transmission data is proportional to the area of the update area determined in principle by the screen update area determination process (B721 and B725). Therefore, the proportionality coefficient can be obtained in advance from the past transmission situation.

  Therefore, the Web server 720 calculates the value of d-2α-Δb1 during the processing of B721 and transmits it to the Web browser 710 using the response D723. After the response is interpreted by the Web browser 710, d-2α-Δb1 When the next screen data request A 712 is transmitted later, screen data can be transmitted and received efficiently using the transmission path.

  The screen capture processing unit 118 of FIG. 1 returns time interval information until the next screen data request calculated by the above method and image data based on the capture result as a response to the Web browser side. At this time, the former time interval information is passed as an HTTP header or cookie of image data.

  In the Web browser of the remote terminal 130, the HTML / Java Script document 120 in the HTML cache 134 is analyzed and executed, so that the browser display processing unit 133 converts the passed image data into the object on the bottom of the drawing layer object 135. Assigned to, move to the foreground and display. Furthermore, the transmission timing of the next image data request is dynamically changed by changing the HTML document in the HTML cache 134 based on the time interval passed by the HTTP header or the like.

  Through the above flow, transmission / reception and display of image data between the device 110 and the remote terminal 130 can be realized.

  On the other hand, in the case of the two-request method, the HTTP server function unit 119 of the device 110 that has received a screen update request from the Web browser of the remote terminal 130 calls the CGI program 117 that is a request handler. The corresponding CGI program 117 determines from the CGI call parameter whether the screen update request is a full screen update or only a drawing area update. If it is a full screen update, the area to be captured is fixed to the full screen, so area calculation is not performed. When only the drawing area is targeted, the screen capture processing unit 118 refers to the drawing area history data 112 held by the HMI screen drawing unit 115 and includes the minimum drawing area including the accumulated drawing area from the previous request. The coordinates of are calculated.

  Note that the amount of image data to be transmitted later can be known from the rectangular information in this area, and the next capture timing to be transmitted from the Web browser can be calculated from the amount of data.

  FIG. 10 is a diagram illustrating a method for determining the screen update request interval time in the case of the two-request method for “efficient use of the transmission path”.

  As described above, the Web browser 810 operates on the remote terminal 130, and the Web server 820 operates on the device 110. Also, A811 and A813 are screen update area requests, B821 and B825 are screen update area confirmation processing, C822 and C826 are screen update area notifications and the time interval until the next request is specified, D812 and D814 are screen data requests, E823 and E827 are Screen capture processing, F824 and F828 are set as screen data transmission processing.

  Now, let α be the transmission delay time between the Web browser 810 and the Web server 820 in the example of this figure. Further, the time required for the screen update area determination processing B821 and B825 is represented by Δa. Further, the time required for the screen capture processing E823 and E827 is assumed to be Δb1 and Δb2, respectively. Further, let d be the time taken for the screen data transmission F824.

  The interval between the first screen update area request A 811 and the second screen update area request A 813 in FIG. 10 is specified so that the next screen data transmission F 828 starts immediately after completion of the screen data transmission F 824. Thus, the efficiency of the transmission line is maximized. The following formulates the interval between the response C822 reception and the request A813 transmission in the Web browser 810 at this time.

  The time interval between screen update area requests is designated to the web browser 810 by C822, which is the first response from the web server 820. Since the value specified at this time needs to be the time from the timing at which the response C822 is interpreted by the Web browser 810, the time in the range of G829 in FIG.

From the above conditions, the following equation (1) is established.
Formula (1): G829 + A813 + B825 + C826 + D814 + E827 = D812 + E823 + F824

When a variable is assigned with the time of G829 as x, the following equation (2) is obtained.
Formula (2): x + α + Δa + α + α + Δb2 = α + Δb1 + d

Therefore, the following expression (3) is obtained.
Formula (3): x = d−2α−Δa + Δb1−Δb2

Here, since the value of Δb2 is unpredictable at the time of B821, the Web server 820 that processes this calculation formula assumes that Δb2 = Δb1, and further ignores the screen update area determination processing (B821 and B825). Since it is completed in as much time as possible, assuming Δa = 0, the following equation (4) is obtained.
Formula (4): x = d-2α

  Further, the value of the screen data transmission time d (an example of the required transmission time) is proportional to the amount of transmission data, and the amount of transmission data is proportional to the area of the update area determined in principle by the screen update area determination process (B821 and B825). Therefore, this proportionality coefficient can be obtained from the past transmission situation.

  Therefore, the Web server 820 calculates the value of d-2α during the processing of B821 and transmits it to the Web browser 810 using the response C822. After the response is interpreted by the Web browser 810, the next screen is displayed after d-2α. By transmitting the update request A813, screen data can be transmitted and received efficiently using the transmission path.

  The screen capture processing unit 118 and the CGI program 117 in FIG. 1 then return a JavaScript for dynamically changing the script in the HTML cache 134 of the remote terminal 130 as a response to the screen update request.

  In the JavaScript text returned to the Web browser as a response, the screen capture request is transmitted, and the time interval value until the next screen update request transmission is rewritten.

  As a result, a screen capture request is transmitted from the Web browser immediately after this, so the CGI program 117 requests the screen capture processing unit 118 to specify the target area and perform capture. Image data based on the capture result is returned as a response to the Web browser.

  In the Web browser of the remote terminal 130, the HTML / Java Script document 120 in the HTML cache 134 is analyzed and executed, so that the browser display processing unit 133 converts the passed image data to the object at the back of the drawing layer object 135. Allocate and move to the foreground before displaying. The next screen update request is transmitted according to the transmission timing of the screen update request that has been changed.

  Through the above flow, transmission / reception and display of image data between the device 110 and the remote terminal 130 can be realized.

  As described above, the change area screen transmission method and display method to the remote terminal 130 according to the present embodiment reduce the transmission data amount when the device 110 having the HMI screen receives a capture request, A screen capture process is performed at high speed, and the image data is transmitted to the remote terminal 130. Furthermore, the image data transmission size is monitored, and the next capture request time from the remote terminal 130 can be obtained so as to have the maximum transmission efficiency at the time of image data transmission. Further, the image data transmitted from the device 110 in this way is displayed on the remote terminal 130 using only a Web browser.

The remote monitoring method between the device 110 having the HMI screen and the remote terminal 130 using the following methods (1) to (5) has been described above.
(1) The drawing process for the HMI screen is recorded on the device 110 side, and when receiving the capture request from the remote terminal 130, the drawing target area is extracted for the drawing process performed from the previous capture request. Capture area determination method for determining a minimum one area including all these areas, and capture method for capturing only a specified area (2) Size of update area transmitted from device 110 side to remote terminal 130 side Then, a capture request time interval determination method for calculating the time required for image data transmission and determining the time until the next capture request from the remote terminal 130. (3) As a capture method for the designated region, the capture target region size is Sampling once for multiple pixels without performing full capture for each pixel if the threshold is exceeded Capture method for performing capture (4) As a capture method for a designated area, when the communication performance with the remote terminal 130 and the browser display speed performance are equal to or lower than a preset threshold value, a plurality of captures are performed without performing full capture for each pixel. Capturing method that performs sampling capture of one pixel per pixel (5) When image data of a partial area of the screen is transmitted to the client side, a plurality of drawing objects are prepared in the client-side browser, A continuous image display method in a browser that repeatedly superimposes the captured image pasted on the drawing object that is hidden and invisible to the user, and repeatedly moves to the foreground

  100 screen transmission system, 110 device, 111 input command event queue, 112 drawing area history data, 113 input command processing unit, 114 main control unit, 115 HMI screen drawing unit, 116 HMI screen data, 117 CGI program, 118 screen capture processing Part, 119 HTTP server function part, 120 HTML / Java Script document, 121 server screen update part, 122 update data transmission part, 130 remote terminal, 131 HTTP client function part, 132 HTML / Java Script analysis part, 133 browser display processing part, 134 HTML cache, 135 drawing layer object, 136 update data reception unit, 137 terminal screen update unit, 141 HTTP request, 142 HTTP response 901 LCD, 902 keyboard, 903 mouse, 904 FDD, 905 CDD, 906 printer, 911 CPU, 912 bus, 913 ROM, 914 RAM, 915 communication board, 920 HDD, 921 operating system, 922 window system, 923 program group, 924 files.

Claims (8)

In a terminal device that outputs a terminal screen configured to superimpose a plurality of layers capable of displaying an image and hide an area where an image displayed on a front layer overlaps with a back layer,
An update data receiving unit for receiving update data indicating display contents after update of an area to be updated in the terminal screen;
Each time update data is received by the update data receiving unit, one layer is selected from the plurality of layers in an arbitrary order, and the display content indicated by the update data is an area in the selected layer, The updated data is displayed in an area corresponding to the area indicating the display content, and the terminal screen is updated by superimposing the plurality of layers so that the selected layer is in front of the previously selected layer, and the updated terminal A terminal device, comprising: a terminal screen update unit that outputs a screen to a display device.   The update data receiving unit receives update data from a server device that updates a server screen capable of displaying an image and transmits update data indicating display contents of an area updated in the server screen, and receives the update data a plurality of times. The terminal device according to claim 1, wherein full-screen data indicating display contents of the entire server screen is received as update data at least once.   The terminal apparatus according to claim 2, wherein the update data receiving unit receives full screen data as update data at least once before the layer selection by the terminal screen update unit is completed. A terminal device that outputs a terminal screen configured to superimpose a plurality of layers that can display an image and hide an area in which the image displayed on the front layer overlaps the back layer, and a server that can display the image In a screen transmission system comprising a server device for updating a screen,
The server device
A server screen update unit for updating the server screen;
An update data transmission unit for generating update data indicating display contents of the area updated by the server screen update unit in the server screen, and transmitting the generated update data to the terminal device;
The terminal device
An update data receiving unit that receives update data transmitted by the update data transmitting unit from the server device;
Each time update data is received by the update data receiving unit, one layer is selected from the plurality of layers in an arbitrary order, and the display content indicated by the update data is an area in the selected layer, The updated data is displayed in an area corresponding to the area indicating the display content, and the terminal screen is updated by superimposing the plurality of layers so that the selected layer is in front of the previously selected layer, and the updated terminal A screen transmission system comprising: a terminal screen update unit that outputs a screen to a display device. The update data receiving unit transmits request data for requesting transmission of update data to the server device,
The update data transmission unit receives the request data transmitted from the terminal device by the update data reception unit, and transmits the update data to the terminal device every time the request data is received.
The update data transmission unit further updates the display data of the area based on the size of the area updated in the server screen each time the server screen is updated by the server screen update unit. The transmission time is calculated as the required transmission time, and based on the calculated transmission required time, the terminal device is instructed to transmit the next request data after receiving the update data. The screen transmission system according to claim 4.   The update data transmission unit performs predetermined sampling on display contents of an area updated by the server screen update unit in the server screen, instead of update data, to the terminal device based on a predetermined condition. 6. The screen transmission system according to claim 4, wherein the sampling data is transmitted. In a terminal program that outputs a terminal screen configured to superimpose a plurality of layers capable of displaying an image and hide an area where an image displayed on the front layer overlaps on the back layer,
Update data reception processing for receiving update data indicating display contents after update of the area to be updated in the terminal screen;
Each time update data is received by the update data reception process, one layer is selected from the plurality of layers in an arbitrary order, and the display content indicated by the update data is an area in the selected layer, The updated data is displayed in an area corresponding to the area indicating the display content, and the terminal screen is updated by superimposing the plurality of layers so that the selected layer is in front of the previously selected layer, and the updated terminal A terminal program that causes a computer to execute terminal screen update processing for outputting a screen to a display device. A server device that updates a server screen capable of displaying an image transmits update data indicating display contents of an area updated in the server screen,
A terminal device that outputs a terminal screen configured to superimpose a plurality of layers capable of displaying an image and hide an area in which the image displayed on the front layer overlaps on the back layer is transmitted from the server device Received update data,
Each time update data is received, the terminal device selects one layer from the plurality of layers in an arbitrary order, and the display content indicated by the update data is an area in the selected layer, and The updated terminal screen is displayed by updating the terminal screen by superimposing the plurality of layers so that the update data is displayed in an area corresponding to the area indicating the display content, and the selected layer is in front of the previously selected layer. Is output to a display device.

Images