TWI845445B - System and method of training symbol recognition model and system and method of detecting display device - Google Patents

Abstract

The present disclosure provides a method of training symbol recognition model, performed by a computing device, and includes: outputting image control signals to a display device and controlling a camera to photograph the display device to obtain training images, using each of the training images as a target image, cutting the target image to obtain symbol image blocks according to default display areas of the display device, assigning a target label to each of the symbol image blocks, wherein the target label includes a first sub-label associated with one of the image control signals corresponding to the target image and a second sub-label indicating a valid status or an invalid status, and using the symbol image blocks and the target labels to performing training to generate a symbol recognition model. The present disclosure further provides a method of detecting display device using the symbol recognition model.

Description

Symbol recognition model training method and system and display device detection method and system

The present invention relates to a symbol recognition model training method and system and a display device detection method and system.

In order to detect whether the display devices used in the factory are abnormal, many factories use optical character recognition (OCR) to identify the content displayed on the display device. The user selects the area to be compared and sets the similarity threshold for detection.

However, since there are multiple display areas on the display device, the above detection method is very time-consuming and may produce erroneous detection results due to manual frame selection errors. In addition, the existing optical character recognition uses image features such as pattern, distance, color, etc. to detect whether the target character is a number, text or special symbol. This method is easily affected by ambient light and requires frequent adjustment of sensitivity or tolerance to compensate for the recognition result.

In view of the above, the present invention provides a symbol recognition model training method and system and a display device detection method and system to solve the above problems.

According to an embodiment of the present invention, a symbol recognition model training method includes executing, by a computing device: outputting a plurality of image control signals to a display device and controlling a camera to shoot the display device to obtain a plurality of frames of training images respectively corresponding to the plurality of image control signals; using each of the plurality of frames of training images as a target image and executing a marking procedure, the marking procedure including: segmenting the target image according to a plurality of preset display areas of the display device to obtain A plurality of symbol image blocks are obtained; and a plurality of target labels are respectively assigned to the plurality of symbol image blocks, wherein the plurality of target labels each include a first sub-label and a second sub-label, the first sub-label is associated with one of the plurality of image control signals corresponding to the target image, and the second sub-label indicates a valid state or an invalid state; and the plurality of symbol image blocks and the plurality of target labels are used for training to generate a symbol recognition model.

A display device detection method according to an embodiment of the present invention is executed by a computing device, including: outputting an image control signal to the display device and controlling a camera to shoot the display device to obtain a test image corresponding to the image control signal; segmenting the test image according to multiple preset display areas of the display device to obtain multiple symbol image blocks; inputting the multiple symbol image blocks into a symbol recognition model to obtain multiple prediction labels of the multiple symbol image blocks, wherein the multiple prediction labels each include a first sub-label and a second sub-label, the first sub-label indicates a predicted symbol, and the second sub-label indicates a valid state or an invalid state; and generating a detection result of the display device according to the degree of matching between the image control signal and the multiple prediction labels.

In summary, the symbol recognition model training method and system according to one or more embodiments of the present invention can automatically collect training data, and the trained symbol recognition model can accurately detect whether the display function of the display device in actual operation is abnormal. In addition, the display device detection method and system according to one or more embodiments of the present invention can effectively detect whether the display device is abnormal.

The above description of the disclosed content and the following description of the implementation methods are used to demonstrate and explain the spirit and principle of the present invention, and provide a further explanation of the scope of the patent application of the present invention.

The detailed features and advantages of the present invention are described in detail in the following embodiments, and the contents are sufficient to enable any person skilled in the relevant art to understand the technical contents of the present invention and implement them accordingly. Moreover, according to the contents disclosed in this specification, the scope of the patent application and the drawings, any person skilled in the relevant art can easily understand the relevant purposes and advantages of the present invention. The following embodiments are to further illustrate the viewpoints of the present invention, but are not to limit the scope of the present invention by any viewpoint.

Please refer to FIG1, which is a block diagram of a symbol recognition model training system according to an embodiment of the present invention. As shown in FIG1, the symbol recognition model training system 1 includes a computing device 10, a display device 11 and a camera 12. The computing device 10 is connected to the display device 11 and the camera 12.

The computing device 10 pre-stores a plurality of preset display areas of the display device 11 and is used to control the camera 12 to capture the display screen of the display device 11, uses the plurality of preset display areas to process the images captured by the camera 12, and uses the processed images for training to generate a symbol recognition model, wherein the symbol recognition model is used to predict the symbol displayed by the display device and whether the symbol is a valid or invalid symbol.

The computing device 10 may include one or more processors and communication elements. The processor may be, for example, a central processing unit, a graphics processor, a microcontroller, a programmable logic controller, or other processors with signal processing functions. The communication element is used for the computing device 10 to communicate with the display device 11. For example, the computing device 10 may include an infrared controller, and the display device 11 may include an infrared receiver; or, the computing device 10 and the display device 11 may each include a wireless communication element such as a Bluetooth element or a wireless network (Wi-Fi); or, the computing device 10 and the display device 11 may each include a wired connection port for interconnection and communication. The display device 11 may be, for example, a liquid crystal display, a light-emitting diode display, etc. having the above-mentioned communication element. Taking the application scenario as an example, the display device 11 can be a display in a factory for displaying processing information. The display device 11 is controlled by the computing device 10 to display images. Furthermore, the display device 11 can be placed in a dark box, and the camera 12 can shoot the display screen of the display device 11 in the dark box to obtain a clearer image for the computing device 10 to perform model training.

In order to explain the method of generating a symbol recognition model in more detail, please refer to FIG. 1, FIG. 2 and FIG. 3, wherein FIG. 2 is a flow chart of a symbol recognition model training method according to an embodiment of the present invention, and FIG. 3 is a schematic diagram showing segmentation of a target image according to a preset display area. As shown in FIG. 2, the symbol recognition model training method includes: step S101: outputting a plurality of image control signals to a display device and controlling a camera to shoot the display device to obtain a plurality of training images corresponding to the plurality of image control signals; using each of the training images as a target image to execute steps S103 and S105, wherein step S103 is: segmenting the target image according to a plurality of preset display areas of the display device to obtain a plurality of symbol images; Image blocks, and step S105 is: assigning multiple target tags to the multiple symbol image blocks respectively, wherein the multiple target tags each include a first sub-tag and a second sub-tag, the first sub-tag is associated with one of the multiple image control signals corresponding to the target image, and the second sub-tag indicates a valid state or an invalid state; and performing step S107: using the multiple symbol image blocks and the multiple target tags for training to generate a symbol recognition model. The steps shown in FIG2 can be executed by the symbol recognition model training system 1 of FIG1, in particular, by the computing device 10. The following exemplarily illustrates the steps of FIG2 by taking the operation of the symbol recognition model training system 1 of FIG1 as an example.

In step S101, the computing device 10 outputs a plurality of image control signals to the display device 11, and controls the camera 12 to shoot the image displayed by the computing device 10 to obtain a plurality of corresponding frames of training images. Specifically, one image control signal may correspond to one frame of training image. The image control signal may be used to control the backlight of the display device 11 and the displayed text, symbols, etc.

Next, the computing device 10 uses each frame of the training image as a target image and executes a marking procedure, wherein the marking procedure includes steps S103 and S105. In step S103, the computing device 10 divides the target image according to a plurality of preset display areas of the display device 11 to obtain a plurality of symbol image blocks, wherein each of the plurality of preset display areas can be an area for displaying one of a seven-segment display, English letters, an electronic dial, a unit, a communication interface diagram, and punctuation marks. A symbol image block can include a symbol, and the symbol can be a number, a word, a punctuation mark, a pattern, and a combination thereof, etc., which is not limited by the present invention.

One of the multiple training images obtained in step S101 may be the target image IMG1 as shown in FIG. 3 , and since the computing device 10 pre-stores multiple preset display areas of the display device 11 , the computing device 10 may segment the target image IMG1 accordingly to generate multiple symbol image blocks A1 - A11 .

In step S105, the computing device 10 assigns a target tag corresponding to each of the symbol image blocks A1-A11, wherein the target tag includes a first sub-tag and a second sub-tag. The computing device 10 assigns the first sub-tag and the second sub-tag corresponding to the symbol image block according to the image control signal. The first sub-tag is the symbol code indicated by the image control signal, and the second sub-tag is the valid state or invalid state indicated by the image control signal. As shown in FIG3 , the symbol image block A1 may correspond to the display area of the seven-segment display, the symbol image blocks A2 and A3 may correspond to the display area of the English letters, the symbol image block A4 may correspond to the display area of the demand period (EOI), the symbol image block A5 may correspond to the display area of the alternate mode (ALT), the symbol image block A6 may correspond to the display area of the punctuation marks, and the symbol image block A7 may correspond to the electronic turntable ( table), the symbol image block A8 may correspond to the display area of the interface icon, such as RS485, infrared and other communication icons, the symbol image block A9 may correspond to the display area of the continuous (Continue, Cont), the symbol image block A10 may correspond to the display area of the cumulative demand (Cumulative, Cum), and the symbol image block A11 may correspond to the display area of the unit icon. Each symbol image block and the corresponding label are shown in Table 1 below, where P represents the second label indicating a valid state, F represents the second label indicating an invalid state, and the numbers and/or English letters following P/F are the first labels.

Table 1 Symbol image block Target label Symbol image block A1 P7058, P7069, F7001～F7057 Symbol image block A2 PA～PD Symbol image block A3 PA～PF Symbol image block A4 PEOI Symbol image block A5 PALT Symbol image block A6 PP0, PP1, PP2, PP3 Symbol image block A7 PT077, PT178, FT001～FT076 Symbol image block A8 PI1～PI4 Symbol image block A9 PCont Symbol image block A10 PCum Symbol image block A11 PU090, PU165, FU001～FU089

Taking symbol image block A1 as an example, if the first sub-tag is the code "7058" and the second sub-tag is in a valid state, then a valid number is displayed in symbol image block A1. In addition, taking symbol image block A1 as an example, assuming that the first sub-tag is the code "7002" and the second sub-tag is in an invalid state, garbled characters may be displayed in symbol image block A1. In other words, if the image control signal controls a preset display area to display a symbol that may appear in the area, the corresponding second sub-tag is in a valid state; conversely, if the image control signal controls a preset display area to display a symbol that will not appear in the area (for example, garbled characters), the corresponding second sub-tag is in an invalid state.

In addition, the computing device 10 can name multiple folders using each symbol and the corresponding target label, and store each symbol image block and the corresponding target label in the corresponding folder to more effectively classify the training data. It should be particularly noted here that FIG. 3 and Table 1 exemplarily present a variety of display areas and labels, but their types can be increased, decreased or changed according to actual needs, and the present invention is not limited thereto. In one embodiment, one or more of the image control signals issued by the computing device 10 can instruct the display device 11 not to display the picture, and assign a label that the image captured by the camera 12 is not displayed on the screen, such as NA.

In step S107, the computing device 10 uses the symbol image blocks and corresponding target labels of each frame of training image for training to generate a symbol recognition model. The symbol recognition model can be learned using transfer. During training, the pre-trained model (i.e., the trained model) is connected between the input layer and the classifier. The input layer is the length, width, and number of channels of the training image, and the output of the classifier is a label in the same form as the target label (for example, the predicted label described later). The pre-trained model can be a convolutional neural network (CNN) model, such as MobileNet-v2. The weights of the input layer and the classifier can be updated accordingly during training.

In addition, before step S107, the computing device 10 can adjust the multi-frame training images according to at least one random image parameter to generate a multi-frame expanded image, and use each of the expanded images as a target image and execute a marking process (step S103 and step S105). The at least one random image parameter includes at least one of brightness, saturation, and contrast. Accordingly, more training images can be obtained without executing steps S101 to S105 multiple times.

Since the symbol recognition model training method according to the present invention collects all the text, numbers, symbols, etc. that may be displayed on the display device 11 and assigns valid and invalid labels, the symbol recognition model generated by the above training can accurately judge the text, symbols, graphics, etc. displayed on the display screen of the display device 11 in actual operation or other display devices that are the same as the display device 11, and can be used to further confirm whether the actual display result conforms to the preset display result corresponding to the control signal to determine whether the display function is abnormal.

Please refer to FIG. 1, FIG. 4 and FIG. 5, wherein FIG. 4 is a flowchart of display screen area capture in a symbol recognition model training method according to an embodiment of the present invention, and FIG. 5 is a schematic diagram of obtaining a display screen area. The step of FIG. 4 is executed before step S101 of FIG. 2, and can be executed by the symbol recognition model training system 1 of FIG. 1, especially by the computing device 10. As shown in FIG. 4 , the method for obtaining a display screen area includes: step S201: controlling a display device to display a preset symbol and controlling a camera to shoot the display device to obtain an initial image; step S203: correcting the initial image to generate a corrected image; and step S205: obtaining a display screen area from the corrected image according to a preset display position and a preset size of the preset symbol.

In step S201, the computing device 10 controls the display device 11 to display a default symbol DS, and controls the camera 12 to capture the display screen of the display device 11 to obtain an initial image. As shown in FIG5, the default symbol DS may be "7". The execution method of step S201 may be similar to step S101 in FIG2.

In step S203 , the computing device 10 calibrates the initial image to generate a calibrated image IMG. The computing device 10 may rotate the initial image so that at least one side of the calibrated image IMG is parallel to a corresponding side of the display device 11 .

In step S205, the computing device 10 obtains the display screen area DA from the calibrated image IMG according to the auxiliary frame DA1 and the default size of the default symbol DS, wherein the default display area DA includes the display screen area. The computing device 10 may obtain the display screen area according to a plurality of default ratios of the coordinate difference between the plurality of coordinates of the default symbol DS in the calibrated image and the default size of the default symbol DS.

The following is an explanation of step S205 with reference to FIG5. The computing device 10 can determine that the default symbol DS includes a plurality of reference points C2 and C3 according to the image control signal, such as the lower right corner and the upper left corner of "7", wherein the coordinates of the reference point C2 are (x2, y2), and the coordinates of the reference point C3 are (x3, y3). The computing device 10 calculates the horizontal coordinate difference w (i.e., x2-x3) between the reference point C2 and the reference point C3, and calculates the vertical coordinate difference h (i.e., y2-y3) between the reference point C2 and the reference point C3, and obtains the coordinates (x4, y4) of the reference point C4 according to the following formula (1), and obtains the coordinates (x5, y5) of another reference point C5 according to the following formula (2), and then obtains the display screen area DA according to the reference point C4 and the reference point C5. R1, R2, R3 and R4 in formulas (1) and (2) are multiple preset ratio values, which can be obtained by formulas (3) to (6), wherein shiftx indicates the preset length between the reference point C1 and the reference point C3 of the default symbol DS, and shifty indicates the preset length between the reference point C1 and the reference point C2 of the default symbol DS. For example, the preset ratio values R1, R2, R3 and R4 can be set to 4.22, 1.23, 6.28 and 1.75, respectively. Formula 1) Formula (2) Formula (3) Formula (4) Formula (5) Formula (6)

By the above method of obtaining the display screen area, the computing device 10 can divide the target image and obtain the symbol image block more accurately. It should be particularly noted that FIG. 5 shows that the default symbol DS can be "7" by way of example, but in other embodiments, the default symbol DS can be other symbols including the reference points C2 and C3.

Please refer to FIG. 1 , FIG. 5 and FIG. 6 , wherein FIG. 6 is a flowchart of image correction in a symbol recognition model training method according to an embodiment of the present invention. The steps of FIG. 6 are executed before step S103 of FIG. 2 , and can be regarded as a detailed flowchart of an embodiment of step S203 of FIG. 4 , and can be executed by the symbol recognition model training system 1 of FIG. 1 , especially by the computing device 10 . As shown in FIG. 6 , the method for correcting an image includes: step S301: obtaining first coordinates and second coordinates of two reference points in an initial image; step S303: obtaining a first angle according to the first coordinates and the second coordinates; and step S305: rotating the initial image based on the difference between the first angle and the reference angle to obtain a corrected image.

It should be noted that, similar to the above, the computing device 10 can determine that the default symbol includes two reference points according to the image control signal, and the connection line between the two reference points corresponds to the frame of the display device 11. Taking FIG. 5 as an example, the default symbol DS includes two reference points C1 and C2, and the connection line between the reference points C1 and C2 is parallel to the short frame of the display device 11 (or/and perpendicular to the long frame of the display device 11). In addition, the reference points C2 and C3 can also be used as the two reference points of the default symbol DS. The following is an explanation using the reference points C1 and C2 as the two reference points of the default symbol DS.

In step S301, the computing device 10 obtains the first coordinates (x1, y1) and the second coordinates (x2, y2) of the reference points C1 and C2 in the initial image. Specifically, the computing device 10 extends outward from the center point of the initial image according to the length and width of the initial image at a predetermined ratio to obtain the auxiliary frame DA1. The predetermined ratio is, for example, 25%, which is not limited in the present invention. Then, the computing device 10 captures the first coordinates (x1, y1) and the second coordinates (x2, y2) of the outline in the auxiliary frame DA1.

In step S303, the computing device 10 obtains a first angle according to the first coordinate and the second coordinate. The computing device 10 may first calculate the slope m of the second coordinate relative to the first coordinate based on the following formula (7), and then convert the slope m into the first angle based on the following formula (8): . Formula (7) Formula (8)

In step S305, the computing device 10 rotates the initial image based on the difference between the first angle and the reference angle to obtain a corrected image. The reference angle is, for example, The computing device 10 may obtain an angle difference between the base angle and the first angle, and rotate the initial image by the angle difference to obtain a corrected image. If the angle difference is a negative value, the initial image is rotated clockwise; if the angle difference is a positive value, the initial image is rotated counterclockwise.

Please refer to FIG. 7, which is a block diagram of a display device detection system according to an embodiment of the present invention. As shown in FIG. 7, the display device detection system 2 includes a computing device 20, a display device 21 and a camera 22. The computing device 20 is connected to the display device 21 and the camera 22.

The computing device 20 is used to control the camera 22 to shoot the display screen of the display device 21, and to use the image shot by the camera 22 to determine whether the display device 21 is abnormal. The computing device 20 may include one or more processors, such as a central processing unit, a graphics processor, a microcontroller, a programmable logic controller or other processors with signal processing functions. The display device 21 is the same type of display device as the display device 11 of FIG. 1. The display device 21 is controlled by the computing device 20 to display images. The computing device 20 and the computing device 10 of FIG. 1 may be the same computing device or different computing devices, and the display device 21 and the display device 11 of FIG. 1 may be the same display device or different display devices.

To explain the display device detection method in more detail, please refer to FIG. 7 and FIG. 8 , wherein FIG. 8 is a flow chart of the display device detection method according to an embodiment of the present invention. The steps shown in FIG. 8 are executed by the display device detection system 2 of FIG. 7 , and in particular, by the computing device 20 . As shown in FIG8 , the display device detection method includes: step S401: outputting an image control signal to the display device and controlling a camera to shoot the display device to obtain a test image corresponding to the image control signal; step S403: segmenting the test image according to a plurality of preset display areas of the display device to obtain a plurality of symbol image blocks; step S405: inputting the plurality of symbol image blocks into a symbol recognition model to obtain a plurality of prediction labels of the plurality of symbol image blocks, wherein the plurality of prediction labels each include a first sub-label and a second sub-label, the first sub-label indicates a predicted symbol, and the second sub-label indicates a valid state or an invalid state; and step S407: generating a detection result of the display device according to a matching degree between the image control signal and the plurality of prediction labels.

In step S401, the computing device 20 outputs an image control signal to the display device 21 and controls the camera 22 to shoot the display device 21 to obtain a test image corresponding to the image control signal. As mentioned above, the display device 21 includes a plurality of preset display areas, and the image control signal output by the computing device 20 is used to control the preset display area to display a symbol that may or may not be presented in the area.

In step S403, the computing device 20 obtains the symbol image block according to the default display area of the display device 21. Step S403 is the same as step S103 of FIG. 2 and may also include the embodiments of FIG. 4 and/or FIG. 5.

In step S405, the computing device 20 inputs the symbol image block into the symbol recognition model to obtain a predicted label of the symbol image block, wherein the symbol recognition model may be a symbol recognition model generated according to the symbol recognition model training method and system described in one or more of the above embodiments. Each predicted label includes a first sub-label and a second sub-label, the first sub-label indicating a code corresponding to the predicted symbol, and the second sub-label indicating whether the code of the predicted symbol is valid or invalid. In other words, the first sub-label generated by the symbol recognition model is used to predict the symbol displayed by the symbol image block, and the second sub-label generated by the symbol recognition model is used to predict whether the symbol is a valid symbol or an invalid symbol.

In step S407, the computing device 20 generates the detection result of the display device 21 according to the matching degree between the image control signal and the predicted label. Taking the symbol image block A1 of FIG3 as an example, if the image control signal indicates that the symbol displayed by the symbol image block A1 is "5", the first sub-label generated by the symbol recognition model is the code of the symbol "5", and the second sub-label indicates a valid state, then the matching degree between the image control signal corresponding to the symbol image block A1 and the predicted label is high. If the image control signal indicates that the symbol displayed by the symbol image block A1 is "5", and the first sub-label generated by the symbol recognition model is a garbled code, and the second sub-label indicates an invalid state, then the matching degree between the image control signal corresponding to the symbol image block A1 and the predicted label is low. Similarly, if the image control signal indicates that the symbol displayed by the symbol image block A1 is garbled, the first sub-label generated by the symbol recognition model indicates that the predicted symbol is a garbled code, and the second sub-label indicates an invalid state, then the image control signal corresponding to the symbol image block A1 has a high degree of match with the predicted label.

In other words, the prediction tag is used to indicate whether the symbol displayed in the test image is consistent with the symbol indicated by the image control signal. If not, it means that the display function of the display device may be abnormal. Accordingly, the above display device detection method and system can effectively detect whether the display device is abnormal.

Please refer to FIG. 7, FIG. 9 and FIG. 10(a) to FIG. 10(b) together, wherein FIG. 9 is a flowchart of the border detection in the display device detection method according to an embodiment of the present invention, FIG. 10(a) is a schematic diagram showing a blank test image and a border detection block, and FIG. 10(b) is a diagram showing the detection result of FIG. 10(a). The steps of FIG. 9 can be executed by the display device detection system 2 of FIG. 7, and in particular, by the computing device 20. As shown in FIG. 9 , the border detection method of the display device includes: step S501: outputting a blank image signal to the display device and controlling a camera to shoot the display device to obtain a blank test image corresponding to the blank image signal; step S503: dividing the blank test image according to a preset width to obtain a plurality of border detection blocks; step S505: respectively calculating the plurality of A plurality of grayscale values of the border detection area; step S507: determining whether each of the plurality of grayscale values is higher than a corresponding threshold; if the determination result of step S507 is "yes", executing step S509: outputting a positive border detection result; and if the determination result of step S507 is "no", executing step S511: outputting a negative border detection result.

In step S501, the blank image signal output by the computing device 20 is used to control the display device 21 to display the backlight without displaying any symbols. The computing device 20 controls the camera 22 to shoot the display device 21 to obtain a blank test image.

Taking FIG. 10( a) as an example, in step S503, the computing device 20 divides the blank test image into the first border detection block BDX1 to the nth border detection block BDXn, where n is a positive integer greater than 1. The width of each of the first border detection block BDX1 to the nth border detection block BDXn may be equal to a preset width.

In step S505, in the examples of FIG. 10(a) and FIG. 10(b), the computing device 20 calculates the grayscale value of each of the first border detection block BDX1 to the nth border detection block BDXn. Since the first border detection block BDX1 and the nth border detection block BDXn both correspond to the entire border of the display device 21, the grayscale values of both are relatively high, while the second border detection block BDX2 corresponds to a partial border of the display device 21, so the grayscale value is relatively low.

In step S507, the computing device 20 determines whether each grayscale value is higher than the corresponding threshold value, wherein different border detection blocks may correspond to different threshold values. Taking FIG. 10(a) and FIG. 10(b) as examples, the threshold values of the first border detection block BDX1 and the nth border detection block BDXn are higher than the threshold value of the second border detection block BDX2. For example, the threshold values of the first border detection block BDX1 and the nth border detection block BDXn may be 200, and the threshold value of the second border detection block BDX2 may be 20.

Taking the second border detection block BDX2 as an example, if the computing device 20 determines that the grayscale value of the second border detection block BDX2 is not higher than the threshold value, it means that the second border detection block BDX2 does not display other content. Therefore, in step S509, the computing device 20 outputs the positive border detection result, wherein the positive border detection result can indicate that the position of the second border detection block BDX2 and its display function are normal.

Taking the second border detection block BDX2 as an example, if the computing device 20 determines that the grayscale value of the second border detection block BDX2 is higher than the threshold value, it means that the second border detection block BDX2 may display other content indicated by the non-blank image signal and has a higher grayscale value. Therefore, in step S511, the computing device 20 outputs a negative border detection result, wherein the negative border detection result can indicate that the position of the second border detection block BDX2 and its display function are abnormal.

Please refer to FIG. 10(c) and FIG. 10(d). FIG. 10(c) is a schematic diagram showing a blank test image and a border detection block, and FIG. 10(d) is a diagram showing the detection result of FIG. 10(c). In addition to detecting the border detection block obtained by segmentation along the horizontal axis as shown in FIG. 10(a) and FIG. 10(b), the border detection block obtained by segmentation along the vertical axis as shown in FIG. 10(c) and FIG. 10(d) can also be detected. The detection method described in FIG. 9 can also be applied to FIG. 10(c) and FIG. 10(d). The details are not repeated here.

According to the above border detection method, even if the display device has only a slight defect, the existence and location of the defect can be effectively detected.

It should also be noted that the detection method shown in FIG. 8 can be executed before or after the detection method shown in FIG. 9 , and the present invention is not limited thereto.

In summary, according to the symbol recognition model training method and system of one or more embodiments of the present invention, training data can be automatically collected, and the trained symbol recognition model can accurately detect whether the display function of the display device in actual operation is abnormal. In addition, by expanding the amount of training data, the convergence speed of the model during training can be improved. According to the display device detection method and system of one or more embodiments of the present invention, it is possible to effectively detect whether the display device is abnormal. In addition, according to the image segmentation method of one or more embodiments above, the outer frame of the symbol can be automatically selected.

Although the present invention is disclosed as above with the aforementioned embodiments, it is not intended to limit the present invention. Any changes and modifications made within the spirit and scope of the present invention are within the scope of patent protection of the present invention. Please refer to the attached patent application for the scope of protection defined by the present invention.

1: Symbol recognition model training system 2: Display device detection system 10,20: Computing device 11,21: Display device 12,22: Camera A1,A2: Symbol image block IMG0: Initial image IMG1: Target image C1,C2,C3,C4,C5: Reference point DA: Display screen area DA1: Auxiliary frame BDX1,BDX2,BDXn,BDY1,BDYn: Border detection block S101,S103,S105,S107,S201,S203,S205,S301,S303,S305,S401,S403,S405,S407,S501,S503,S505,S507,S509,S511: Steps

FIG. 1 is a block diagram of a symbol recognition model training system according to an embodiment of the present invention. FIG. 2 is a flow chart of a symbol recognition model training method according to an embodiment of the present invention. FIG. 3 is a schematic diagram showing segmentation of a target image according to a preset display area. FIG. 4 is a flow chart of display screen area capture in a symbol recognition model training method according to an embodiment of the present invention. FIG. 5 is a schematic diagram showing acquisition of a display screen area. FIG. 6 is a flow chart of image correction in a symbol recognition model training method according to an embodiment of the present invention. FIG. 7 is a block diagram of a display device detection system according to an embodiment of the present invention. FIG8 is a flow chart of a display device detection method according to an embodiment of the present invention. FIG9 is a flow chart of a border detection in a display device detection method according to an embodiment of the present invention. FIG10(a) to FIG10(d) are schematic diagrams corresponding to FIG9.

S101, S103, S105, S107: Steps

Claims (12)

A symbol recognition model training method includes executing by a computing device: Output multiple image control signals to a display device and control a camera to shoot the display device to obtain multiple frames of training images corresponding to the image control signals; use each of the training images as a target image and execute a labeling procedure, the labeling procedure comprising: segmenting the target image according to multiple preset display areas of the display device to obtain multiple symbol image blocks; and assigning multiple target labels to the symbol image blocks respectively, wherein the target labels each include a first sub-label and a second sub-label, the first sub-label is associated with one of the image control signals corresponding to the target image, and the second sub-label indicates a valid state or an invalid state; and use the symbol image blocks and the target labels for training to generate a symbol recognition model. The symbol recognition model training method as described in claim 1 further includes: Controlling the display device to display a preset symbol and controlling the camera to shoot the display device to obtain an initial image; correcting the initial image to generate a corrected image; and obtaining a display screen area from the corrected image according to the preset display position and preset size of the preset symbol; wherein the preset display areas include the display screen area. The symbol recognition model training method as described in claim 2, wherein obtaining the display screen area from the calibrated image according to the default display position and the default size of the default symbol comprises: Obtaining the display screen area according to the coordinate difference between multiple coordinates of the default symbol in the calibrated image and multiple default ratios of the default size. A symbol recognition model training method as described in claim 2, wherein the default symbol includes two reference points, the line between the two reference points corresponds to a border of the display device, and correcting the initial image to generate the corrected image includes: Obtaining a first coordinate and a second coordinate of the two reference points in the initial image; obtaining a first angle based on the first coordinate and the second coordinate; and rotating the initial image based on the difference between the first angle and a reference angle to obtain the corrected image. The symbol recognition model training method as described in claim 1 further comprises before the training: Adjusting the training images according to at least one random image parameter to generate multi-frame extended images; and using each of the extended images as the target image and executing the labeling process; wherein the at least one random image parameter includes at least one of brightness, saturation and contrast. A symbol recognition model training system includes the computing device, the display device and the camera for executing the symbol recognition model training method as described in any one of claims 1 to 5. A display device detection method, executed by a computing device, includes: outputting an image control signal to a display device and controlling a camera to shoot the display device to obtain a test image corresponding to the image control signal; segmenting the test image according to multiple preset display areas of the display device to obtain multiple symbol image blocks; inputting the symbol image blocks to a symbol recognition model to obtain multiple prediction labels of the symbol image blocks, wherein the prediction labels each include a first sub-label and a second sub-label, the first sub-label indicates a predicted symbol, and the second sub-label indicates a valid state or an invalid state; and generating a detection result of the display device according to the matching degree between the image control signal and the prediction labels. The display device detection method as described in claim 7 further comprises: Controlling the display device to display a preset symbol and controlling the camera to shoot the display device to obtain an initial image; correcting the initial image to generate a corrected image; and obtaining a display screen area from the corrected image according to the preset display position and preset size of the preset symbol; wherein the preset display areas include the display screen area. The display device detection method as described in claim 8, wherein obtaining the display screen area from the calibration image according to the default display position and the default size of the default symbol comprises: Obtaining the display screen area according to the coordinate difference between multiple coordinates of the default symbol in the calibration image and multiple default ratios of the default size. A display device detection method as described in claim 8, wherein the preset symbol includes two reference points, the line between the two reference points corresponds to a border of the display device, and correcting the initial image to generate the corrected image includes: Obtaining a first coordinate and a second coordinate of the two reference points in the initial image; obtaining a first angle based on the first coordinate and the second coordinate; and rotating the initial image based on the difference between the first angle and a reference angle to obtain the corrected image. The display device detection method as described in claim 7 further comprises: Outputting a blank image signal to the display device and controlling the camera to shoot the display device to obtain a blank test image corresponding to the blank image signal; dividing the blank test image according to a preset width to obtain a plurality of border detection blocks; respectively calculating a plurality of gray values of the border detection blocks; and outputting a negative border detection result when any of the gray values is higher than a corresponding threshold value. A display device detection system includes the computing device for executing the display device detection method as described in any one of claims 7 to 11, the display device and the camera.

Images