WO2022023831A1

WO2022023831A1 - Smart display application with potential to exhibit collected outdoor information content using iot and ai platforms

Info

Publication number: WO2022023831A1
Application number: PCT/IB2021/055353
Authority: WO
Inventors: Mahdi AKHAVAN
Original assignee: Akhavan Mahdi
Priority date: 2020-07-29
Filing date: 2021-06-17
Publication date: 2022-02-03

Abstract

Upon using IoT technologies and artificial intelligence (AI), the invention aims to lower the rate of instantaneous error of image processing using CRM, as well as displaying contents appropriate to the style of preferences and interests of the majority of outdoor users by detailed analyzing outdoor information using sensor. In this plan, the advertiser will first intelligently receive the outdoor information collected via the system panel, and then based on the received information and its display conten, will record several conditions for intelligent display in the system. After displaying the content, the outdoor users will be polled about the broadcast content, and the users will be able to register their comments with defined gestures (symbols). Ultimately, the angles of the audiences' heads and eyes in the environment will be calculated using the image-processing technology and relative to the display and the relevant data will be displayed in the advertisers' panel.

Description

Title of Invention : Smart Display application with potential to exhibit collected Outdoor Information Content Using loT and Al Platforms

Technical Field

[0001] The technical field of the invention relates to electronic, loT technology, artificial intelligence, machine learning, image, and audio processing.

Background Art

[0002] KR1020180078475 - INTERNET-OF-THINGS SMART ADVERTISEMENT DISPLAY APPARATUS CAPABLE OF REMOTE CONTROL AND METHOD THEREOF

[0003] The remotely controllable loT-based smart advertising display was filed in the patent request number KR1020180078475. This invention relates to an loT- based smart display characterized by an advertising content storage module and shows advertisements based on the infrared module and relevant information processing. At the same time, it applies a beacon module for signal transfer.

[0004] The similarity of the present invention to the claimed one is content to display in terms of user activity processing; however, information is collected from the infrared module in the invention, whereas processing is based on the camera, microphone, voice, temperature, light, and many other parameters in the claimed invention.

[0005] Another invention (JP2017162221A) entitled Information Processing Device, Content Evaluation, and Program which was filed in 2017 in Japan, proposed an advertisement display system equipped with memory, an imaging system, and an image processing system. After the advertising content is displayed, the system starts capturing a video. At the same time, the image processing system acts as a facial and motion recognition system to count the number of individuals who stop and watch the advertisement when it starts to display. The system also counts the number of individuals who pass indifferently to analyze the effectiveness of the content. Moreover, the facial recognition system analyzes people’s satisfaction based on their facial expressions. Their faces are then stored in memory with a specific ID for every advertisement. Therefore, it is possible to determine what content was appropriate enough to draw people’s attention. According to the analysis results, the stored information is limited only to people’s faces, and CRM has no role in error reduction. Audio, temperature, and humidity sensors were not used. The sound of an advertisement is not processed through outdoor information. People’s words are not retrieved. It is also impossible to limit the imaging frames.

Technical Problem

[0006] The current system is applied to accurately identify user's faces and retrieve their information. Given the high error rates in real-time image processing and user-camera distance, the previous systems were prone to user recognition errors. Therefore, outdoor information might not be effective for displaying the appropriate advertising or educational content. Moreover, real-time image processing might not provide other relevant information such as occupation, education, purchase history, and annual income of the user. Furthermore, previous methods were focused on using image processing for displaying the contents. However, in this invention as well as the image, the outdoor sound, light, temperature, and humidity are also processed to select the most relevant content and information.

[0007] The outside advertising system, now used in most countries, is usually offered to clients without any consultations or based on vague estimations. As clients don’t have access to accurate information on the surrounding area, they have to select an environment based on the reported statistics. However, the selected site might not yield the desired turnover for their business. Besides, clients must consult with different companies and individuals for each advertising site, something that might require clients to incur more costs and spent more time.

[0008] In the current systems, advertisers are unable to realize how many users have seen the advertisements and how well-liked it has been among the users. [0009] Non-cost effectiveness, limited choices of display sites, and lack of continuous changing the outside advertising contents have led the clients to prefer other advertising methods.

Solution to Problem

[0010] Once the display is turned on, content is displayed first based on what is stored in the memory. At the same time, the central controller monitors the outdoor data through the camera, microphone, temperature, and humidity sensors. After processing these data, it sends the outdoor information to the cloud server. Furthermore, this information will continuously be sent to and stored at the cloud server online.

[0011] After registering in the system and uploading the properties of their video and image contents, advertisers can receive some smart recommendations from the system based on the collected outdoor data. Definition of specific conditions for their contents, an advertiser can then present their contents on displays by showing advertisements at a specific time (First method) or showing advertisements based on the instant processing of outdoor data (Second method). Once confirmed by an expert, the uploaded content is transmitted online from the cloud server to displays.

[0012] In the first method, the advertiser sets the time and location for content display; thus, the content will automatically be displayed at a specific time, whereas in the second method, it will be displayed based on conditions defined by an advertiser and processing the retrieved outdoor data. In both cases of content display, the number of people seeing the content will be counted. They will also be asked about the displayed content. Finally, the resultant information will be stored and displayed on the content owner’s panel. The number of views of the displayed advertisement is measured through the image processing technology from the head and eye angles of viewers who are present in the environment.

[0013] The head and eye angles of audiences are determined through the machine learning technology implemented by defining a neural network. The necessary training data of the network are collected from the images captured by outdoor cameras in a few crowded places and are then used to train the network after the labeling process. After the network is trained with a huge data, the output network model is used to detect the head and eye angles of the audiences present in the environment.

[0014] The invention matches the displayed content with the preferences and interests of most users in the environment to improve the efficiency of this method. Due to the limited display duration of every advertising or educational content in this invention, different businesses have more opportunities. For instance, the display duration of every advertising content will be 30 seconds, which can allow for the display of 120 advertisements in an hour. This can give different businesses more opportunities and reduce the costs of outdoor advertisements.

[0015] Advertisers can also benefit from a feature called " pay as you go ". Using this feature, advertisers define the display conditions of their campaign and then run it. With the image processing technology, this feature determines the head and eye angles of the audiences present in the environment concerning the display. Accordingly, advertisers will be charged in their accounts.

[0016] Nowadays, the outdoor advertising system is often used in most countries with no counseling for advertisers that must also negotiate with advertising companies to register their advertisements in any place at any time. However, this invention benefits from an artificial intelligence system to automatically provide an advertiser with some recommendations based on the collected outdoor data of any environment (e.g. the general traffic data of people and vehicles classified based on gender, clothing style, age, the number of people, type of vehicles, and other parameters collected from the predefined sensors and presented to the advertiser).

[0017] To improve the accuracy of the collected data (mentioned in Technical Problems), the system first analyzes people’s faces in the customer relationship management (CRM) database when processing the images captured by the camera connected to the board (raspberry pi). The system then sends all user information that has already existed in the CRM along with the instantly effective parameters to the server to display the efficient content. If user information is unavailable in databases, user demographic information (e.g. height, age, gender, weight, etc.), people’s clothing, number of people, their presence duration, their facial expressions (e.g. sad, happy, energetic, etc.), skin color, hair color, eye color, people’s placement time and location in the environment, and information of objects (e.g. duration of existence, the dominant brand in the environment, object dimensions, dominant color, brightness and outdoor light, number of objects, license plate detection, object coverage, placement of objects, movement of objects, sound intensity, etc.) are processed at the moment to show content on displays in proportion to the retrieved information.

[0018] Drawing upon deep learning, and image processing algorithms, people’s features such as age, gender, and duration of presence are classified by the Landmark algorithm. The GrabCut algorithm is used to classify people’s clothing. Furthermore, human body dimensions, facial expressions (e.g. sad, happy, energetic, etc.) are detected through the support vector machine algorithm (SVM) and the OpenCV library, whereas FCM and clustering algorithms are used to detect skin color, hair color, eye color, and dominant environment colors. In addition, the FIVT-FIMM model is utilized to determine people’s placement time and location and their movements in an environment, whereas convolutional neural networks are used to determine the existence time of objects and the dominant brand in an environment.

[0019] To measure object dimensions, counting the number of objects and number of people, detect license plates, object covers, and dominant colors the YOLO algorithm and deep learning algorithms are applied. The placement and movement of objects are determined through the OpenCV library and Tensorflow library.

[0020] The CRM information (Figure 8) can be registered through the following methods:

[0021] 1- Information is stated by users as registering on a website or application.

[0022] 2- Applying user face processing information is retrieved from other information systems such as shops, healthcare centers, and public centers. [0023] 3- Information available on the system and about the previous user traffic can be analyzed and registered (e.g. the system creates a user account automatically and stores the information for a user who passes the displays in different areas. After that, the user attributed information will be stored in the user account wherever the user is seen again. In addition, the user will be able to see and complete the information by registering in the system).

[0024] The CRM stores different pieces of information as educational degrees, age, marital status, demographics, income per capita, religion, occupation, race, height, weight, skin color, hair color, eye color, social relationships, nationality, place of residence, educational records, type of clothing, banking information, social media accounts, purchase records, number of views, and purchases from shops. The CRM system is also able to connect to all the systems which can be connected through API, a feature that updates and validates the user inserted information constantly.

[0025] Facial recognition is performed on the present individuals separately.

According to Figure 11 , the information is processed, and the system will finally show content based on the outdoor information and predefined conditions.

[0026] Upon processing the outdoor voices, the system analyzes the dominant words and sends the output information to the cloud server. This information includes dialects and languages. Using this feature, the system can display content based on a combination of dominant voices and images in the environment. In other words, the advertisements will be shown based on the words uttered by the users present in the environment. This feature also allows for the smart volume adjustment of displays. The SpeechRecognition is used to convert outdoor voices into texts and process words by adjusting the display volume.

[0027] Temperature and humidity sensors process outdoor data and send them to the cloud server via the central board (raspberry pi).

[0028] A light sensor (GY-30) is applied to transmit outdoor brightness to the server; therefore, an advertiser can display their contents accordingly by defining their conditions. The display brightness is also adjusted smartly in this way, something which will reduce energy consumption. [0029] The number of sensors differs in different places. For instance, it is unnecessary to use the voice sensor on the highway; however, all sensors are used in a public place like a train station so that the system can process the environment properly.

[0030] To improve the accuracy of the image processing (Figure 9), a content owner may adjust the image processing regions based on the distance between a camera and individuals as well as the camera quality by determining those regions while defining the content display conditions.

[0031] Defining the conditions, an advertiser can also specify whether users are processed and analyzed when they are moving or standing to show the content accordingly.

[0032] In their panels, advertisers can define the loT-based object connection information and conditions for use in the smart content display.

[0033] In the direct display method, the advertiser should set some conditions for content display at different times and places after registering the contents; however, they define more conditions for the more purposive presentation of contents on displays by using the capabilities of outdoor sensors (Figure 11) in the smart display method They can also be defined campaign condition based on the Fuzzy algorithm.

[0034] As there are wireless connections between cameras and the controller board, image processing cameras can be placed within one or several kilometers away from a display in order to improve the accuracy and decision-making of advertising presentations on highways. The content is then displayed after the information is processed on a display connected to the central board.

[0035] Facial expressions and hand gestures of the audiences were processed to receive their feedback on the displayed content (Figure 13). This method detects certain reactions such as frown, smile, surprise, and indifference and reports them to the system. Users can also send their feedback to the displayed content with hand gestures such as liking, disliking, and showing indifference (a horizontal like). The support vector machine algorithm (SVM) and the OpenCV library were used for facial expression recognition, whereas the convolutional neural network algorithm was used for hand gesture detection.

[0036] The invention can be connected to online TV channels to display content such as an instructive message that might say, “Wash your hands and face every hour to prevent the COVID-19!” and show the important results of a sports game.

[0037] The invention can also connect to different displays such as TVs, projectors, big screens, smartphones, virtual reality glasses, and tablets to show the content transmitted by the cloud server. Furthermore, the smartphones, tablets, and smart TVs designed for direct display need no hardware boards and can only be connected to the system if the necessary application is installed on them.

Advantageous Effects of Invention

[0038] - Interactive display content: The invention can be used to process outdoor data such as voices, images, light, and temperature and display the relevant content in proportion to user preferences in those areas to draw more attention and finally improve efficiency and branding.

[0039] - The high number of views: Using image and head angle processing, this invention can count the number of views of every product after an advertisement is displayed.

[0040] - Knowing how many times an advertisement is displayed: The advertiser can see the number of times an advertisement is displayed at different times in the dashboard panel.

[0041] - Voice processing: This feature can be used to display outdoor advertisements in proportion to the dominant keywords and conditions set by advertiser and to adjust the display speakers based on the outdoor sound.

[0042] - Image processing: through using “cameras” connected to the board this invention allows the system to process data of objects as well as the preferences and demographics of people in the place. The data are then sent to the cloud server, which will then generate the command to display advertising programs online in proportion to the retrieved data and the conditions defined by advertiser. Due to the errors existing in the instant image processing, this invention first processes and analyzes user faces in the CRM system. If the user information exists in the CRM system, the relevant content will be shown by using the registered information and combining it with other instantly effective parameters such as user faces, user clothing, voices, and temperature and processing the data of objects existing in the environment. If the user information does not exist in the system, online image processing will be used to retrieve identifiable user demographics based on their faces (gender, height, age, weight, etc.), motion detection, clothing styles, voices, light, temperature, and data of objects (brand, properties and dimensions of objects, placement of objects, license plate detection, etc.) existing in the environment. The content will then be displayed based on the retrieved information and defined conditions. For this group of users, user accounts will be created in the CRM automatically.

[0043] - Showing advertisements based on outdoor temperature: Using this feature, an advertiser can show an advertisement in proportion to the outdoor temperature by defining certain conditions while registering the advertisement.

[0044] - Showing advertisements based on light intensity: Using this feature and defining conditions while registering the advertisement, an advertiser can show the advertisement to users in proportion to the outdoor light intensity. The display brightness is also adjusted based on the outdoor light intensity, something which will reduce the display energy consumption.

[0045] - Smart counseling: Analyzing outdoor data and user data, this invention can provide advertisers with useful information to produce content or advertisements. Advertisers will then create their advertisements accordingly and use the invention to show them on integrated displays.

[0046] - Centralizing and integrating the smart advertising system and reducing outdoor advertising costs: In this invention, the proposed method is the integrated presentation of contents on different displays. This helps display owners avoid extra costs for software licenses, software support, and hardware support. Advertisers can also present their advertisements on the displays of different areas in smart and direct methods based on the smart system recommendations.

[0047] - Showing advertisements on different displays: This invention can be connected to all modern displays (LCD, LED, and OLED TV sets, video projectors, VR glasses, smartphones, tablets, computers, etc.) to show advertising contents and information.

[0048] - Connectivity to loT sensors: This invention was designed on an loT platform; therefore, it is able to communicate with loT-based sensors. This kind of connection is also called m2m or machine-to-machine connection.

[0049] - Extendibility and connectivity to other information systems via API: This invention can be connected to all API-based systems for information exchange. Information and banking systems include accounting departments of shops, warehouse systems of products, banking information, rapid alert systems, etc.

[0050] - Incurring costs for advertisement presentation (pay as you go): The invention allows advertisers to pay for the costs based on views and attention of outdoor users.

[0051] - Defining conditions: In the direct presentation method, the advertiser should set some conditions for content presentation after registering the content. These conditions include presentation time, content presentation location, etc. However, in the smart presentation method, advertisers employ the capabilities of outdoor sensors to define more purposive conditions for their contents on displays so that the presented content will be efficient enough for the audiences present in an environment.

[0052] - High security: All connections between displays, sensors, and the cloud server are coded in end-to-end encryption, something which will increase the system security.

[0053] - CRM information bank system: This system is among the most important components of this invention. It is used to register and constantly update the information of all users who use the system. If the information is entered into the system wrongly, the system will detect and correct the errors by accessing other information systems and processing outdoor information and user information. The CRM information is entered into the system by users or other information systems. The information includes user demographics, educational degrees, age, marital status, income per capita, religion, occupation, race, height, weight, social relationships, skin color, hair color, eye color, nationality, place of residence, banking information, social media accounts, purchase records, views of a shop, and purchases made. If this information is entered correctly, users will see smarter and more relevant outdoor information.

[0054] - Surveying after showing every advertisement: At the end of every advertisement or instructive content, users can state their opinions on the presented content by showing like or dislike symbol with their hand.

[0055] - Smart presentation of local news networks: This system can be connected to all public and official systems (e.g. local and international TV networks) of a country to show important announcements and news.

[0056] - Online streaming: Using this feature, all displays retrieve and show data online. If there is no connection to the cloud server, the relevant content will be presented based on the data stored on the memory, and after reconnecting to the server, all gather data will be transfer

Brief Description of Drawings

[0057] FIG (1 ): the general schematic of the smart display through the analysis of the collected data

[0058] FIG (2): the smart display flowchart

[0059] FIG (3): the relay circuit (connecting and disconnecting the utility power)

[0060] FIG (4): the schematic of temperature and humidity sensors

[0061] FIG (5): the schematic of camera detection to the ep8266 board

[0062] FIG (6): the schematic of microphone detection to the esp32 board

[0063] FIG (7): the schematic of lux meter detection to the esp8266

[0064] FIG (8): collect user information

[0065] FIG (9): boundaries of image analysis

[0066] FIG (10): loT-based object connections to smart displays

[0067] FIG (11): defining conditions based on the predefined sensors

[0068] FIG (12): defining conditions to show the smart content

[0069] FIG (13): analysis of user reactions [0070] FIG (14): shows the smart advertising display in proportion to the retrieved outdoor information

[0071] FIG (15): shows the smart advertising display in proportion to the retrieved outdoor information

Description of Embodiments

[0072] This invention is made up of software and hardware components. The software component consists of specific panels for advertisers, displayers, relevant experts, and ordinary users in the system as well as image, voice, temperature, humidity, and light processing panels. The hardware component consists of a central board, a temperature sensor, a humidity sensor, a relay (to connect and disconnect power), a cloud server, displays, cameras and microphones, and a lux meter. Interconnected, all of these components are connected to the cloud server via the central controller (raspberry pi). After the outdoor data are processed, all commands are sent from the cloud server to the display based on the predefined conditions (Figure 1 -7).

[0073] 1- Advertiser

[0074] 2- Ordinary user

[0075] 3- Connections to other information systems via API

[0076] 4- Connects to loT-based sensors

[0077] 5- Cloud server

[0078] 6- loT communication network

[0079] 7- Central board (raspberry pi)

[0080] 8- Display

[0081] 9- First Method of Data Collection [0082] 10- Second Method of Data Collection [0083] 11- Third Method of Data Collection

[0084] 12- Information Systems (e.g. National Organization for Civil Registration) [0085] 13- IoT-based objects such as the products accumulated in a warehouse or shelves of a shop

Examples

[0086] Using this invention, advertisers can present their advertisements publicly in crowded areas at very low costs. Animated and intriguing videos and shows contents based on the analysis of outdoor information will improve the efficiency of this advertising method.

[0087] Advertisers can define the information for connection to loT-based objects in their panels and define certain conditions for the presentation of smart content.

[0088] For instance, a shop manager (Figure 10) can use the RFID tag for every product and register all of its specifications in the chip memory. If a product is to expire (for example, within three days), the loT-based RFID chip will automatically connect to adjacent displays based on the defined conditions and announces to the outdoor audiences near the shop that this product is going to be sold with a specific discount percentage. This will significantly decrease the extra costs of shops and will increase the business owner’s income.

[0089] In the direct presentation method, advertisers must define some conditions for content presentation at different times and places after registering the content. However, in the smart presentation method, advertisers use the features of outdoor sensors (Figure 11 ) to define more conditions for the more purposive presentation of their contents on displays. These conditions can be defined based on both the input data and the fuzzy algorithm.

[0090] Example 1 - An automotive company (Figure 12) can use the dominant brand and license plate as the content presentation base in the content convention definition to direct users to the nearest sales representative. For instance, SAIPA can define the following conditions to show the advertisements of its new product “Shahin” on displays:

[0091] 1- Cars should be stopped.

[0092] 2- The stopping time should be more than 15 seconds.

[0093] 3- The dominant brand is SAIPA. [0094] 4- License plates belong to Tehran and Karaj (Iran plates) (Example : 20, 10, 99, 88, 77, 66, 55, 44, 33, 22, 11 -68-21-78).

[0095] 5- From 12 to 18 o’clock.

[0096] Example 2 - Based on the outdoor information processing (Figure 14), smart displays show users the appropriate content for an environment. For instance, if the weather is sunny, a pair of sunglasses will be advertised. If it is rainy, an umbrella or a cafeteria nearby the display will be advertised. All contents will be shown based on the conditions defined by advertisers and outdoor information processing.

[0097] Example 3 - The advertiser of a dress manufacturer and shop would like to show content at specific hours and places nearby their branches based on gender, age, and the dominant outdoor color.

[0098] If the dominant population includes women mostly aged 20-35 years old in front of the displays between 18 and 20 in a mostly blue environment, a blue dress will be shown on displays (Figure 15).

Industrial Applicability

[0099] Example 2 - Based on the outdoor information processing (Figure 14), smart displays show users the appropriate content for an environment. For instance, if the weather is sunny, a pair of sunglasses will be advertised. If it is rainy, an umbrella or a cafeteria nearby the display will be advertised. All contents will be shown based on the conditions defined by advertisers and outdoor information processing.

[0100] Example 3 - The advertiser of a dress manufacturer and shop would like to show content at specific hours and places nearby their branches based on gender, age, and the dominant outdoor color.

[0101] If the dominant population includes women mostly aged 20-35 years old in front of the displays between 18 and 20 in a mostly blue environment, a blue dress will be shown on displays (Figure 15).

Claims

[Claim 1] The smart display application with the potential to exhibit collected outdoor information comprising: a. Connecting to the CRM information to lower the image processing errors in the identification of outdoor users b. Information processing and analysis system using the camera, microphone, temperature, humidity, and light sensors c. Using the loT-based inventory information of an advertiser for content display

[Claim 2] According to Claim 1 , the processing and analysis system uses different pieces of information pertaining to demographics, occupation, religion, nationality, education, annual income, purchase records of customers, dress code, duration of presence in an environment, user’s distance from the display or the camera, people’s movements, skin color, hair color, eye color, and height to show content s intelligently based on the conditions set by the advertiser.

[Claim 3] According to Claim 1 , the image processing system displays the dimensions of the most relevant content by retrieving information from outdoor sensors and identifying certain variables such as dominant brands, similar objects, and duration of presence, license plate, dominant color, brightness, object-camera distance, and object movements.

[Claim 4] According to Claim 1 , the information processing and analysis system recognizes the words uttered in an environment and displays the most relevant content.

[Claim 5] According to Claim 1 , the outdoor sound and light intensity are monitored by the system to adjust the content volume and display brightness accordingly.

[Claim 6] According to Claim 1 , the information processing and analysis system analyze the head angles of users towards the display to collect the advertisement review data.

[Claim 7] According to Claim 1 , the information processing and analysis system analyze the hand gestures and head movements of users to determine their response to the displayed content. This information is utilized to measure the audience's satisfaction with the advertised content.

[Claim 8] According to Claim 1 , the information processing and analysis system can narrow down the image processing areas in cameras to improve the accuracy of the retrieved information.

[Claim 9] According to Claim 1 , the information processing and analysis system allow the advertiser to define conditions for the content display based on the information collected from outdoor sensors.