KR20220053549A - Inline malware detection - Google Patents

Abstract

Detection of malicious files is initiated. A set comprising one or more sample classification models is stored on the networking device. An N-gram analysis is performed on the sequence of received packets associated with the received file. Performing the n-gram analysis includes using at least one stored sample classification model. A determination is made whether the received file is malicious based at least in part on an n-gram analysis of the sequence of received packets. In response to determining that the file is malicious, propagation of the received file is prevented.

Description

Inline malware detection

Malware is a generic term generally used to refer to malicious software (eg, including various hostile, intrusive, and/or other unwanted software). Malware may be in the form of code, scripts, active content, and/or other software. Exemplary uses of malware include disrupting computer and/or network operations, stealing proprietary information (eg, confidential information, such as identity, financial, and/or intellectual property related information), and/or private/proprietary computer and gaining access to systems and/or computer networks. Unfortunately, as technologies are developed to help detect and mitigate malware, unscrupulous authors find ways to circumvent these efforts. Accordingly, there is an ongoing need for improvements to techniques for identifying and mitigating malware.

Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings.
1 illustrates an example of an environment in which malicious applications are detected and prevented from causing harm.
2A illustrates an embodiment of a data device.
2B is a functional diagram of the logical components of an embodiment of a data device;
3 illustrates an example of logic components that may be included in a system for analyzing samples.
4 illustrates portions of an exemplary embodiment of a threat engine.
5 illustrates an example of a portion of a tree.
6 illustrates an example of a process for performing inline malware detection on a data device.
7A illustrates an example hash table for a file.
7B illustrates an example threat signature for a sample.
8A illustrates an example of a process for performing feature extraction.
8B illustrates an example of a process for creating a model.

The present invention is a process; Device; system; composition of matter; a computer program product embodied on a computer readable storage medium; and/or may be implemented in a variety of ways, including a processor, such as a processor stored on and/or configured to execute instructions provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of steps in the disclosed processes may be varied within the scope of the present invention. Unless otherwise stated, a component, such as a processor or memory, that is described as being configured to perform a task may be implemented as a general component temporarily configured to perform a task at a given time, or as a specific component manufactured to perform a task. can As used herein, the term 'processor' refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

The detailed description of one or more embodiments of the invention is provided below in conjunction with the accompanying drawings, which illustrate the principles of the invention. Although the present invention is described in connection with these embodiments, the present invention is not limited to any embodiments. The scope of the invention is limited only by the claims and the invention includes many alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for purposes of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material known in the technical fields related to the present invention has not been described in detail so as not to unnecessarily obscure the present invention.

1. Overview

Firewalls generally protect networks from unauthorized access while allowing authorized communications to pass through the firewall. A firewall is typically a device, set of devices, or software running on a device that provides firewall functionality for network access. For example, a firewall may be integrated into the operating systems of devices (eg, computers, smartphones, or other types of network communication capable devices). A firewall can also be configured with various types of devices, such as computer servers, gateways, network/routing devices (eg, network routers), and data appliances (eg, security appliances or other types of special purpose devices). It may be implemented as or integrated into one or more software applications on the platform, and in various implementations, certain operations may be implemented in special purpose hardware, such as an ASIC or FPGA.

Firewalls typically deny or allow network transmission based on a set of rules. These sets of rules are often referred to as policies (eg, network policies or network security policies). For example, a firewall may filter inbound traffic by applying a set of rules or policies to prevent unwanted external traffic from reaching protected devices. A firewall may also filter outbound traffic by applying a set of rules or policies (eg, allow, block, monitor, notify or log, and/or other actions may be specified in the firewall rules or firewall policies). , which may be triggered based on various criteria, as described herein). A firewall may also filter local network (eg, intranet) traffic by similarly applying a set of rules or policies.

Security devices (eg, security appliances, security gateways, security services, and/or other security devices) provide various security functions (eg, firewall, anti-malware, intrusion prevention/detection, data loss prevention (DLP) ), and/or other security functions), networking functions (eg, routing, quality of service (QoS), workload balancing of network related resources, and/or other networking functions), and/or other functions. can For example, routing functions may be based on source information (eg, IP address and port), destination information (eg, IP address and port), and protocol information.

A basic packet filtering firewall filters network communication traffic by examining individual packets transmitted over the network (eg, packet filtering firewalls or first generation firewalls, which are stateless packet filtering firewalls). Stateless packet filtering firewalls typically inspect individual packets themselves and apply rules based on the inspected packets (eg, using a combination of the packet's source and destination address information, protocol information, and port number). .

Application firewalls may also perform application layer filtering (eg, application layer filtering firewalls or second generation firewalls, operating on the application level of the TCP/IP stack). Application layer filtering firewalls or application firewalls are generally used for specific applications and protocols (eg, Hypertext Transfer Protocol (HTTP), Domain Name System (DNS) requests, file transfer using File Transfer Protocol (FTP), and Telnet; web browsing using different protocols and various other types of applications, such as DHCP, TCP, UDP, and TFTP (GSS). For example, application firewalls may attempt to stealth unauthorized protocols that attempt to communicate over a standard port (eg, an unauthorized protocol that attempts to steal by using a non-standard port that can typically be identified using application firewalls). /out-of-policy protocols) can be blocked.

Stateful firewalls may also perform state-based packet inspection, where each packet is inspected within the context of a series of packets associated with the flow of said network transmission of packets. This firewall technology is generally referred to as a stateful packet because it maintains a record of all connections that pass through the firewall and can determine whether a packet is the start of a new connection, part of an existing connection, or an invalid packet. For example, the state of the connection itself may be one of the criteria that triggers a rule within a policy.

Advanced or next-generation firewalls may perform stateless and stateful packet filtering and application layer filtering as discussed above. Next-generation firewalls may also implement additional firewall technologies. For example, certain newer firewalls, sometimes called advanced or next-generation firewalls, may also identify users and content (eg, next-generation firewalls). In particular, certain next-generation firewalls expand the list of applications that these firewalls can automatically identify to thousands of applications. Examples of such next-generation firewalls are commercially available from Palo Alto Networks, Inc. (eg, PA series firewalls from Palo Alto Networks). For example, Palo Alto Network's next-generation firewalls enable enterprises to: APP-ID for accurate application identification, user-ID for user identification (eg, by user or group of users), and content- for real-time content scanning identify applications, users, and content - not ports, IP addresses, and packets - using a variety of identification techniques, such as ID (eg, to control web surfing and limit data and file transfers); allow you to control These identification techniques allow enterprises to securely enable application usage using business-related concepts, instead of following the conventional approach provided by conventional port-blocking firewalls. Additionally, special-purpose hardware (eg, implemented as dedicated devices) for next-generation firewalls typically provides higher performance levels for application inspection (eg, reducing network throughput while minimizing latency) than software running on general-purpose hardware. (such as security appliances provided by Palo Alto Networks, Inc.) using dedicated, function-specific processing tightly integrated with a single-pass software engine to maximize.

Advanced or next-generation firewalls may also be implemented using virtualized firewalls. Examples of such next-generation firewalls are commercially available from Palo Alto Networks, Inc. (eg, VMware® ESXi™ and NSX™, Citrix® Netscaler SDX™, KVM/OpenStack (Centos/RHEL, Ubuntu®) VM series firewalls from Palo Alto Networks, and Amazon Web Services (AWS)) supporting a variety of commercial virtualized environments, including It can support firewall and advanced threat prevention features, allowing enterprises to securely enable applications flowing into and across their private, public, and hybrid cloud computing environments.VM monitoring, dynamic address groups, and Automation features such as REST-based APIs allow enterprises to actively monitor VM changes that dynamically feed the context into security policies, thereby eliminating policy lag that can occur when VMs change.

II. Exemplary Environment

1 illustrates an example of an environment in which malicious applications (“malware”) are detected and prevented from causing harm. As will be described in greater detail below, malware classifications (eg, as made by security platform 122 ) may be variously shared and/or improved among various entities included in the environment illustrated in FIG. 1 . can Using the techniques described herein, devices, such as endpoint client devices 104-110, can be protected from such malware.

The term “application” is used throughout the specification to collectively refer to programs, bundles of programs, manifestations, packets, etc., regardless of form/platform. An "application" (also referred to herein as a "sample") may be a standalone file (eg, a calculator application with the file name "calculator.apk" or "calculator.exe") and is also an independent component of another application. (eg, a mobile advertising SDK or library embedded within the calculator app).

"Malware" as used in this application refers to an application that participates in behaviors, whether secret (and illegal), whether or not its user is disapproved/will not be disapproved if fully known. Examples of malware include Trojans, viruses, rootkits, spyware, hacking tools, keyloggers, and the like. One example of malware is a desktop application that collects the location of an end user and reports it to a remote server (which does not provide location-based services, such as mapping services, to the user). Another example of malware is a malicious Android Application Package.apk (APK) file that appears to be a free game to the end user, but secretly sends SMS premium messages (eg costing $10 each), increasing the end user's phone bill. am. Another example of malware is the Apple iOS Flashlight application, which secretly collects users' contacts and sends these contacts to spammers. Other forms of malware may also be detected/frustrated using the techniques described herein (eg, ransomware). Not only that, while n-grams/feature vectors/output accumulation variables are described herein as being generated for malicious applications, the techniques described herein also provide profiles for other kinds of applications (eg, adware profiles, goodware profiles, etc.) may be used in various embodiments.

The techniques described in this application can be applied to various platforms (eg, desktops, mobile devices, gaming platforms, embedded systems, etc.) and/or various types of applications (eg, Android .apk files, iOS applications, Windows PE files, Adobe Acrobat PDF files, etc.). In the example environment shown in FIG. 1 , client devices 104 - 108 are (respectively) a laptop computer, a desktop computer, and a tablet residing in the enterprise network 140 . The client device 110 is a laptop computer that resides outside the corporate network 140 .

Data appliance 102 may have policies regarding communications between client devices, such as client devices 104 and 106 , and nodes that are external to enterprise network 140 (eg, reachable via external network 118 ). are designed to implement Examples of such policies are those governing traffic shaping, quality of service, and routing of traffic. Other examples of policies include those that require scanning for threats in incoming (and/or outgoing) email connections, website contacts, files exchanged via instant messaging programs, and/or other file transfers; It contains the same security policies. In some embodiments, data device 102 is also configured to enforce policies for traffic that does not leave corporate network 140 .

An embodiment of a data appliance is shown in Fig. 2a. The example shown is a representation of the physical components included in data device 102 in various embodiments. Specifically, the data device 102 includes a high performance multi-core central processing unit (CPU) 202 and a random access memory (RAM) 204 . Data device 102 also includes storage 210 (such as one or more hard disks or solid state storage units). In various embodiments, data device 102 monitors enterprise network 140 and stores information used to implement the disclosed techniques (RAM 204, storage 210, and/or other suitable locations). regardless). Examples of such information include application identifiers, content identifiers, user identifiers, requested URLs, IP address mappings, policy and other configuration information, signatures, hostname/URL categorization information, malware profiles, and machine learning. include models. Data device 102 may also include one or more optional hardware accelerators. For example, the data device 102 may include a cryptographic engine 206 configured to perform encryption and decryption operations, and one or more field programmable programs configured to perform matching, operate as network processors, and/or perform other tasks. gate arrays (FPGAs) 208 .

Functionality described herein as being performed by data device 102 may be provided/implemented in various ways. For example, data appliance 102 may be a dedicated device or set of devices. The functionality provided by the data appliance 102 may also be integrated on or executed as software on a general purpose computer, computer server, gateway, and/or network/routing device. In some embodiments, at least some services described as being provided by data appliance 102 are instead (or otherwise) provided by software executing on the client device (eg, client device 104 or client device). (110)).

Whenever data device 102 is described as performing a task, a single component, a subset of components, or all components of data device 102 may cooperate to perform the task. Similarly, whenever a component of data device 102 is described as performing a task, the subcomponent may perform the task and/or the component may perform the task in conjunction with other components. . In various embodiments, portions of data device 102 are provided by one or more third parties. Depending on factors such as the amount of computing resources available to the data device 102 , various logical components and/or features of the data device 102 may be omitted and the techniques described herein are adapted accordingly. . Similarly, additional logical components/features may be included in embodiments of data device 102 where applicable. One example of a component included in data device 102 in various embodiments is an application identification engine configured to identify an application (eg, using various application signatures to identify applications based on packet flow analysis). . For example, application identification engines include web browsing - social networking; web browsing - news; You can determine what type of traffic the session carries, such as SSH, etc.

2B is a functional diagram of the logical components of an embodiment of a data device; The illustrated example is a representation of logical components that may be included in data device 102 in various embodiments. Unless otherwise specified, the various logical components of data device 102 are generally implementable in a variety of ways, including as a set of one or more scripts (eg, written in Java, Python, etc., where applicable).

As shown, data device 102 includes a firewall, and includes a management plane 232 and a data plane 234 . The management plane is responsible for managing user interactions, such as by providing a user interface for configuring policies and viewing log data. The data plane is responsible for managing data, such as by performing packet processing and session handling.

Network processor 236 is configured to receive packets from client devices, such as client device 108 , and provide them to data plane 234 for processing. Whenever flow module 238 identifies packets as being part of a new session, it creates a new session flow. Subsequent packets will be identified as belonging to the session based on the flow search. If applicable, SSL decryption is used by SSL decryption engine 240 . Otherwise, processing by the SSL decryption engine 240 is omitted. Decryption engine 240 helps data device 102 inspect and control SSL/TLS and SSH encrypted traffic, thus stopping threats that might otherwise remain cloaked in encrypted traffic. Decryption engine 240 may also help prevent sensitive content from leaving enterprise network 140 . Decryption may be selectively controlled (eg, enabled or disabled) based on parameters such as URL category, traffic source, traffic destination, user, user group, and port. In addition to decryption policies (eg, specifying which sessions to decrypt), decryption profiles may be assigned to control various options for sessions controlled by the policy. For example, the use of certain cryptographic suites and cryptographic protocol versions may be required.

The application identification (APP-ID) engine 242 is configured to determine what type of traffic the session carries. As an example, application identification engine 242 may recognize a GET request in the received data and conclude that the session requires an HTTP decoder. In some cases, such as in a web browsing session, the identified application may change, and such changes will be noted by the data device 102 . For example, a user may initially browse to a corporate wiki (categorized based on URLs visited as "Web Browsing - Productivity") and then to a social networking site (to URLs visited as "Web Browsing - Social Networking"). classified on the basis of). Different types of protocols have corresponding decoders.

Based on the determination made by the application identification engine 242 , the packets are, by the threat engine 244 , put the packets (which may be received out of order) into the correct order, perform tokenization, and extract information transmitted to an appropriate decoder configured to The threat engine 244 also performs signature matching to determine what should happen to the packet. As required, the SSL encryption engine 246 may re-encrypt the decrypted data. Packets are forwarded using forward module 248 for transmission (eg, to a destination).

As also shown in FIG. 2B , policies 252 are received and stored in management plane 232 . Policies may include one or more rules, which may be specified using domain and/or host/server names, where the rules are based on the subscriber/IP based on various extracted parameters/information from monitored session traffic flows. One or more signatures or other matching criteria or heuristics may be used, such as for security policy enforcement on flows. Interface (I/F) forwarder 250 is provided for management communications (eg, via (REST) APIs, messages, or network protocol communications or other communications mechanisms).

III. security platform

Returning to FIG. 1 , assume that a malicious individual (using system 120 ) created malware 130 . A malicious individual hopes that a client device, such as client device 140 , executes a copy of malware 130 , compromising the client device, such as making the client device a bot in a botnet. The compromised client device then performs a task (eg, participating in cryptocurrency mining, or denial of service attacks), if applicable, and reports the information to an external entity, such as a command and control (C&C) server 150 . In addition, it may be instructed to receive instructions from the C&C server 150 .

Assume that data device 102 intercepts an email sent (eg, by system 120 ) to user “Alice” operating client device 140 . A copy of malware 130 was attached to the message by system 120 . As an alternative, but not a similar scenario, data device 102 may intercept the attempted download by client device 140 of malware 130 (eg, from a website). In either scenario, data device 102 determines whether a signature for the file (eg, email access or website download of malware 130 ) is present on data device 102 . The signature, if present, may indicate that the file is known to be safe (eg, whitelisted), and may also indicate that the file is known to be malicious (eg, blacklisted).

In various embodiments, data device 102 is configured to operate in concert with secure platform 122 . As an example, secure platform 122 may provide a set of signatures of known-malicious files (eg, as part of a subscription) to data device 102 . If the signature for malware 130 is included in the set (eg, the MD5 hash of the malware 130 ), the data device 102 can respond accordingly (eg, the MD5 hash of the email connection sent to the client device 140 is the malware). By detecting a match with the MD5 hash of 130 ), transmission of the malware 130 to the client device 104 may be prevented. The security platform 122 also provides a list of known malicious domains and/or IP addresses to the data appliance 102 so that the data appliance 102 can communicate with the enterprise network 140 and the C&C server 150 (eg, a C&C server). (if 150 is known to be malicious) can be allowed to block traffic between them. The list of malicious domains (and/or IP addresses) can also help data device 102 determine when one of its nodes has been compromised. For example, if a client device 140 attempts to contact the C&C server 150, such an attempt is a strong indicator that the client 104 has been compromised by malware (and may cause the client device 104 to contact the enterprise network 140 ). ), corrective actions should be taken accordingly), such as isolating from communicating with other nodes in As will be described in more detail below, the security platform 122 also transmits other types of information to the data device 102 , such as a set of machine learning models that can be used by the data device 102 to perform inline analysis of the files. ) (eg, as part of a subscription).

In various embodiments, various actions may be taken by the data device 102 if no signature for the connection has been found. As a first example, data device 102 may be safety-guaranteed by blocking the transmission of any connections that are not whitelisted as benign (eg, do not match signatures of known good files). The downside of this approach is that when they are in fact benign there can be many legitimate connections that are unnecessarily blocked as potential malware. As a second example, the data device 102 can be a security threat by allowing the transmission of any connections that are not blacklisted as malicious (eg, do not match the signatures of known bad files). A disadvantage of this approach is that newly created malware (not previously seen by platform 122 ) will not be prevented from causing harm.

As a third example, the data device 102 may provide a file (eg, malware 130 ) to the security platform 122 for static/dynamic analysis, determine whether it is malicious, and/or otherwise classify it. can be configured. Various actions may be taken by the data device 102 while analysis by the secure platform 122 of the connection (signature not already present) is being performed. As a first example, data device 102 may prevent email (and attachments) from being forwarded to Alice until a response is received from secure platform 122 . Assuming that platform 122 takes approximately 15 minutes to thoroughly analyze the sample, this means that the incoming message to Alice will be delayed by 15 minutes. In this example, since the attachment is malicious, this delay will not negatively affect Alice. In an alternative example, suppose someone sends a time-sensitive message to Alice with a positive attachment for which the signature also does not exist. Delaying delivery of the message to Alice by 15 minutes will likely be seen as unacceptable (eg, by Alice). As will be described in greater detail below, an alternative approach is to perform at least some real-time analysis of the attachment on the data device 102 (eg, while waiting for a verdict from the platform 122 ). If the data device 102 can independently determine whether the attachment is malicious or benign, it can take an initial action (eg, block or allow delivery to Alice) and, if applicable, determine whether the attachment is malicious or benign. ) may coordinate and/or take additional actions if received from

The secure platform 122 stores copies of the received samples in storage 142 and analysis is initiated (or scheduled, if applicable). An example of the storage device 142 is an Apache Hadoop Cluster (HDFS). The results of the analysis (and additional information about the applications) are stored in database 146 . If the application is determined to be malicious, the data device may be configured to automatically block the file download based on the analysis result. In addition, a signature may be generated and distributed for malware (eg, data devices such as data devices 102 , 136 and 148 ) to automatically block future file delivery requests to download files determined to be malicious. as).

In various embodiments, secure platform 122 is configured with one or more dedicated commercially available hardware servers (eg, multi-core processor(s), 32G+ running conventional server-class operating systems (eg, Linux)). RAM, gigabit network interface adapter(s), and hard drive(s)). The secure platform 122 may be implemented across a scalable infrastructure including a number of such servers, solid state drives, and/or other applicable high-performance hardware. The secure platform 122 may include several distributed components, including components provided by one or more third parties. For example, portions or all of the secure platform 122 may be implemented using Amazon Elastic Compute Cloud (EC2) and/or Amazon Simple Storage Service (S3). Furthermore, as with data device 102 , whenever secure platform 122 is referred to as performing a task, such as storing data or processing data, a sub-component or It will be appreciated that multiple sub-components (whether individually or in cooperation with third party components) may cooperate to perform the task. As an example, security platform 122 may optionally perform static/dynamic analysis in cooperation with one or more VM servers, such as virtual machine (VM) server 124 .

Examples of virtual machine servers include commercially available server-class hardware (eg, multi-core processors, 32+ gigabytes of RAM, and one or more gigabit network interface adapters). In some embodiments, the virtual machine server is omitted. Furthermore, the virtual machine server may be under the control of the same entity that manages the security platform 122 , but may also be provided by a third party. As an example, the virtual machine server may depend on EC2, and the remaining portions of the secure platform 122 are owned by the operator of the secure platform 122 and provided by dedicated hardware under its control. VM server 124 is configured to provide one or more virtual machines 126 - 128 for emulating client devices. Virtual machines may run various operating systems and/or versions thereof. Observed behaviors resulting from running applications on virtual machines are logged and analyzed (eg, for indications that the application is malicious). In some embodiments, log analysis is performed by a VM server (eg, VM server 124 ). In other embodiments, the analysis is performed at least in part by other components of the secure platform 122 , such as the coordinator 144 .

In various embodiments, secure platform 122 makes available results of analysis of samples via a list of signatures (and/or other identifiers) for data device 102 as part of a subscription. For example, security platform 122 may periodically send a content package identifying malware apps (eg, daily, hourly, or some other interval, and/or based on an event configured by one or more policies). . An example content package is a list of identified malware apps, along with information such as a package name, a hash value to uniquely identify the app, and a malware name (and/or malware family name) for each identified malware app. includes The subscription may cover analysis of these files intercepted by the data device 102 and transmitted by the data device 102 to the secure platform 122 , and may also cover any malware (or other form) known to the secure platform 122 . may cover the signatures of a subset of its malware (eg, mobile malware) except for its malware (eg, PDF malware). As will be described in more detail below, the platform 122 may also include machine learning models (eg, via techniques other than hash-based signature matching) that can help the data device 102 detect malware; You can make other types of information available.

In various embodiments, the security platform 122 is configured to provide security services to various entities other than (or instead of, where applicable, the operator of the data device 102 ). For example, other enterprises, with their own respective enterprise networks 114 and 116 , and their own respective data devices 136 and 148 , may contract with the operator of the secure platform 122 . . Other types of entities may also utilize the services of the secure platform 122 . For example, an Internet service provider (ISP) that provides Internet services to the client device 110 may contract with the security platform 122 to analyze applications that the client device 110 attempts to download. As another example, the owner of the client device 110 may install software on the client device 110 in communication with the secure platform 122 (eg, receive content packages from the secure platform 122 , and using the received content packages to examine the attachments according to the techniques described in , and to send the applications to the secure platform 122 for analysis).

IV. Analyzing samples using static/dynamic analysis

3 illustrates an example of logic components that may be included in a system for analyzing samples. The analysis system 300 may be implemented using a single device. For example, the functionality of the analysis system 300 may be implemented in the malware analysis module 112 integrated into the data instrument 102 . The analysis system 300 may also be implemented collectively, across multiple separate devices. For example, the functionality of the analytics system 300 may be provided by the secure platform 122 .

In various embodiments, the analysis system 300 uses lists, databases, or other collections (shown collectively as collection 314 in FIG. 3 ) of known safe content and/or known bad content. Collection 314 may include through subscription services (eg, provided by a third party) and/or as a result of other processing (eg, performed by data device 102 and/or secure platform 122 ). , can be obtained in various ways. Examples of information contained in collection 314 include: URLs, domain names, and/or IP addresses of known malicious servers; URLs, domain names, and/or IP addresses of known secure servers; URLs, domain names, and/or IP addresses of known command and control (C&C) domains; signatures, hashes, and/or other identifiers of known malicious applications; signatures, hashes, and/or other identifiers of known secure applications; signatures, hashes, and/or other identifiers of known malicious files (eg, Android-enabled files); signatures, hashes, and/or other identifiers of known secure libraries; and signatures, hashes, and/or other identifiers of known malicious libraries.

A. Collection

In various embodiments, when a new sample is received for analysis (eg, an existing signature associated with the sample does not exist in the analysis system 300 ), it is added to the queue 302 . As shown in FIG. 3 , the application 130 is received by the system 300 and added to the queue 302 .

B. Static Analysis

The coordinator 304 monitors the queue 302 , and as resources (eg, a static analysis worker) become available, the coordinator 304 fetches samples from the queue 302 for processing (eg, malware (withdraw a copy of 130). In particular, the coordinator 304 first provides a sample to the static analysis engine 306 for static analysis. In some embodiments, one or more static analysis engines are included in analysis system 300 , where analysis system 300 is a single device. In other embodiments, the static analysis is performed by a separate static analysis server comprising a plurality of actuators (ie, a plurality of instances of the static analysis engine 306 ).

The static analysis engine obtains general information about the sample and includes it in the static analysis report 308 (along with heuristics and other information where applicable). The report may be generated by the static analysis engine, or by the coordinator 304 (or by another suitable component), which may be configured to receive information from the static analysis engine 306 . In some embodiments, the collected information is stored in a database record for the sample (eg, in place of or in addition to a separate static analysis report 308 that is generated (ie, portions of the database record from the report 308 ). database 316). In some embodiments, the static analysis engine also forms a verdict (eg, “safe,” “suspicious,” or “malicious”) for the application. As an example, the determination may be “malicious” if one “malicious” static characteristic is present in the application (eg, the application contains a hard link to a known malicious domain). As another example, points may be assigned to each of the features (eg, if found, based on severity; based on how reliable the feature is to predict malice, etc.) and the decision is a point associated with static analysis results. may be assigned by the static analysis engine 306 (or the coordinator 304, if applicable) based on the number of them.

C. Dynamic Analysis

Once the static analysis is complete, the coordinator 304 locates an available dynamic analysis engine 310 to perform a dynamic analysis on the application. Like the static analysis engine 306 , the analysis system 300 may directly include one or more dynamic analysis engines. In other embodiments, the dynamic analysis is performed by a separate dynamic analysis server comprising a plurality of workers (ie, a plurality of instances of the dynamic analysis engine 310 ).

Each dynamic analysis worker manages a virtual machine instance. In some embodiments, the results of the static analysis (eg, performed by the static analysis engine 306 ) are in the form of a report 308 and/or stored in the database 316 , or otherwise stored as such. Regardless, it is provided as input to the dynamic analysis engine 310 . For example, the static reporting information may be used to help select/customize the virtual machine instance used by the dynamic analysis engine 310 (eg, Microsoft Windows 7 SP2 vs. Microsoft Windows 10 Enterprise, or iOS 11.0 vs. iOS 12.0). ). In the case where multiple virtual machine instances are running concurrently, a single dynamic analysis engine may manage all of the instances, if applicable, or multiple motion analysis engines may be used (eg, each with its own virtual machine). instance management). As will be described in more detail below, during the dynamic part of the analysis, actions taken by the application (including network activity) are analyzed.

In various embodiments, static analysis of the sample is omitted or performed by a separate entity, if applicable. As an example, conventional static and/or dynamic analysis may be performed on the files by a first entity. Once a given file is determined to be malicious (eg, by a first entity), the file is then transferred to a second entity (eg, by dynamic analysis engine 310 ) for further analysis (eg, by the dynamic analysis engine 310 ) specifically for the malware's use of network activity. (eg, the operator of the secure platform 122 ).

The environment used by the analytics system 300 is instrumented/hooked (eg, using a customized kernel that supports hooking and logcat) such that behaviors observed while the application is running are logged as they occurred. Network traffic associated with the emulator is also captured (eg, using pcap). The log/network data may be stored as temporary files on the analytics system 300 , and may also be stored more permanently (eg, using HDFS or another suitable storage technology or a combination of technologies, such as MongoDB). The dynamic analysis engine (or another suitable component) compares 314 the connections made by the samples to lists of domains, IP addresses, etc. and whether the sample has communicated with (or attempts to communicate with) malicious entities. did) can be determined.

Like the static analysis engine, the dynamic analysis engine stores the results of its analysis in the database 316 in a record associated with the application being tested (and/or includes the results in the report 312 if applicable). In some embodiments, the dynamic analysis engine also forms a verdict (eg, “safe,” “suspicious,” or “malicious”) for the application. As an example, a determination may be "malicious" if a single "malicious" action is taken by the application (eg, an attempt is made to contact a known malicious domain, or an attempt to leak sensitive information is observed). As another example, points can be assigned to actions taken (eg, based on severity if found; based on how reliable the action is to predict malice; etc.) and the decision is a point associated with dynamic analysis results. may be assigned by the dynamic analysis engine 310 (or the coordinator 304, if applicable) based on the number of them. In some embodiments, a final determination associated with a sample is made based on a combination of report 308 and report 312 (eg, by coordinator 304 ).

V. Inline Malware Detection

Returning to the environment of FIG. 1 , millions of new malware samples can be created each month (eg, by making subtle changes to existing malware or by authoring new malware, such as the operator of system 120 , by immoral individuals). Thus, there will be many malware samples for which the secure platform 122 (at least initially) does not have a signature. Furthermore, even when secure platform 122 has generated signatures for newly generated malware, resource constraints are imposed by data devices, such as data device 102 , on a list of all known signatures at any given time (eg, platform prevent having/using (as stored on 122).

Sometimes malware, such as malware 130 , will successfully penetrate network 140 . One reason for this is if the data device 102 operates according to a “first-time allow” principle. When data device 102 does not have a signature for the sample (eg, sample 130 ) and submits it to secure platform 122 for analysis, secure platform 122 returns a verdict (eg, “benign”). , "malicious", "unknown", etc.) assume it takes about 5 minutes. Instead of blocking communications between the system 120 and the client device 104 during the five minute time period, under the first grant principle, the communications are allowed. When the verdict is returned (eg, after 5 minutes), the data device 102 can use the verdict (eg, “malicious”) to block subsequent transmissions of the malware 130 to the network 140 , and the system Communication between 120 and the network 140 may be blocked. In various embodiments, if the second copy of the sample 130 arrives at the data device 102 during the time period, the data device 102 awaits a determination from the secure platform 122 and the second copy of the sample 130 arrives at the data device 102 . A copy (and any subsequent copies) is maintained by the system 120 to defer the response from the secure platform 122 .

Unfortunately, while the data appliance 102 waits for a verdict from the secure platform 122 for five minutes, the user of the client device 104 could execute the malware 130 , potentially allowing the client device 104 in the network 140 . ) or other nodes. As noted above, in various embodiments, the data device 102 includes a malware analysis module 112 . One task that the malware analysis module 112 may perform is inline malware detection. In particular, and as will be described in greater detail below, as a file (such as sample 130 ) passes through data device 102 , machine learning techniques may be used to perform efficient analysis of the file on data device 102 . (eg, concurrently with other processing performed on the file by the data device 102 ) and an initial maliciousness verdict may be determined by the data device 102 (eg, the security platform 122 ). ) while waiting for a verdict from).

Various difficulties may arise when implementing such analysis on resource constrained devices, such as data devices 102 . One critical resource on device 102 is session memory. A session is a network transfer of information, including files, that device 102 parses according to the techniques described herein. A single device can have millions of concurrent sessions, and the memory available to persist for a given session is extremely limited. A first difficulty when performing inline analysis on a data device such as data device 102 is that, due to these memory constraints, data device 102 will typically not be able to process the entire file at once, and instead As a unit, it is receiving a sequence of packets it needs to process. The machine learning approach used by the data appliance 102 will therefore need to accommodate packet streams in various embodiments. A second difficulty is that, in some cases, data device 102 will not be able to determine where the end of a given file being processed (eg, end of sample 130 in the stream) occurs. The machine learning approach used by the data device 102 is thus, in various embodiments, potentially in an intermediate stream (eg, midway through the reception/processing of the sample 130 or otherwise before the end of the actual file) for a given file. You will be asked to make a decision.

A. Machine Learning Models

As will be described in greater detail below, in various embodiments, the security platform 122 provides to the data device 102 a set of machine learning models for the data device 102 for use with inline malware detection. The models incorporate characteristics (eg, n-grams or other characteristics) determined by the security platform 122 to correspond to malicious files. Two example types of such models include linear classification models and non-linear classification models. Examples of linear classification models that may be used by data appliance 102 include logistic regression and linear support vector machines. An example of a non-formal classification model that may be used by the data appliance 102 includes a gradient boosting tree (eg, eXtreme Gradient Boosting (XGBoost)). The non-linear model is more accurate (and may be better at detecting obfuscated/topological malware), but the linear model uses significantly fewer resources on the device 102 (and efficiently parsing JavaScript or similar files). more appropriate).

As will be described in more detail below, what type of classification model is used for a given file being analyzed may be based on (and, for example, determined by a magic number) the filetype associated with the file.

1. Additional details about the threat engine

In various embodiments, data device 102 includes a threat engine 244 . The threat engine integrates both protocol decoding and threat signature matching during each decoder stage and pattern match stage. The results of the two stages are merged by the detector stage.

When data device 102 receives a packet, data device 102 performs a session match to determine which session the packet belongs to (allowing data device 102 to support concurrent sessions). Each session has a session state that engages a specific protocol decoder (eg, a web browsing decoder, an FTP decoder, or an SMTP decoder). When a file is transmitted as part of a session, the applicable protocol decoder may use an appropriate file-specific decoder (eg, a PE file decoder, a JavaScript decoder, or a PDF decoder).

Portions of an exemplary embodiment of a threat engine 244 are shown in FIG. 4 . For a given session, the decoder 402 walks the traffic in a bytestream, following the corresponding protocol and indicating contexts. One example of a context is an end-of-file context (eg, encountering </script> while processing a JavaScript file). The decoder 402 may indicate an end-of-file context in the packet, which may then be used to trigger execution of the appropriate model using the observed characteristics of the file. In some cases (eg, FTP traffic), explicit protocol-level tags may not be present for the decoder 402 to identify/indicate the context. As will be described in greater detail below, in various embodiments, the decoder 402 determines when feature extraction of the file should end (eg, the overlay section begins) and execution with the appropriate model should begin. Other information (eg, file size as reported in the header) may be used to

The decoder 402 includes two parts. The first part of the decoder 402 is a virtual machine portion 404, which may be implemented as a state machine using a state machine language. The second part of the decoder 402 is a set of tokens 406 (eg, deterministic finite automation (DFA) or regular expressions) for triggering state machine transitions and actions when matched to traffic. The threat engine 244 also includes a threat pattern matcher 408 (eg, using regular expressions) that performs pattern matching (eg, to threat patterns). As an example, threat pattern matcher 408 may be provided with a table of strings to match (whether exact strings or wildcard strings), and corresponding actions to take if a string match is found (eg, security platform 122 ). ) by). Detector 410 processes the outputs provided by decoder 402 and threat pattern matcher 408 to take various actions.

2. N-grams

Data in a session can be decomposed into a sequence of n-grams - a series of byte strings. As an example, suppose a piece of hexadecimal data in a session is: "1023ae42f6f28762aab". The 2-grams in the sequence are all pairs of contiguous letters, such as "1023", "23ae", "ae42", "42f6", etc. In various embodiments, the threat engine 244 is configured to analyze files using 8-grams. Other n-grams may also be used, such as 7-grams or 4-grams. In the example string above, "1023ae42f6f28762" is 8-grams, and "23ae42f6f28762aa" is 8-grams. The total number of different 8-grams possible in a byte sequence is 2 ⁶⁴ (18,446,744,073,709,551,616). Searching for all possible 8-grams in a byte sequence would easily exceed the resources of the data device 102 . Instead, and as will be described in greater detail below, a significantly reduced set of 8-grams is provided by the security platform 122 to the data device 102 for use by the threat engine 244 .

As session packets corresponding to the file are received by the threat engine 244 , the threat pattern matcher 408 parses the packets for matches to strings in a table (eg, a regular expression and/or an exact string). by performing matches). A list of matches (eg, with each instance of the match identified by the corresponding pattern ID) and at what offset each match occurred is generated. Actions for these matches are taken in order of offset (eg, from low to high). For a given match (ie, corresponding to a particular pattern ID), a set of one or more actions to take is specified (eg, via an action table mapping actions to pattern IDs).

The set of 8-grams provided by the security platform 122 may be added (eg, as exact string matches) as additions to the table of matches that the threat pattern matcher 408 is already performing (eg, a JavaScript file). heuristic matches that look for specific indications of malware, such as accessing this password store, or when a PE file calls the Local Security Authorization Subsystem Service (LSASS) API). One advantage of this approach is that instead of performing multiple passes through the packet (eg, evaluating first for heuristic matches and then for 8-gram matches), 8-grams are matched against a threat pattern. It may be searched concurrently with other searches performed by group 408 .

As will be described in more detail below, 8-gram matches are used by both linear and non-linear classification models in various embodiments. Example operations that may be specified for n-gram matches include incrementing a weight counter (eg, for a linear classifier) and storing the match in a feature vector (eg, for a non-linear classifier). do. What action to take may be specified based on the filetype associated with the packet (which determines what type of model is used).

3. Choosing a model

In some cases, a given filetype is specified within the header of the file (eg, as a magic number appearing in the first 7 bytes of the file itself). In such a scenario, the threat engine 244 may select an appropriate model corresponding to the specified file type (eg, based on a table provided by the security platform 122 listing the file types and corresponding models). In other cases, such as JavaScript, a magic number or other filetype identifier (if present in the header) may not test which classification model should be used. As an example, JavaScript would have a file type of "text file". To identify filetypes such as JavaScript, the decoder 402 performs deterministic finite state automation (DFA) pattern matching and uses heuristics (eg, identifying <script> and other indicators that the file is JavaScript). can be used to The determined file type and/or the selected classification model is stored in the session state. The filetype associated with the session may be updated as the session progresses, if applicable. For example, in a stream of text, when encountering a <script> tag, a JavaScript filetype can be assigned for a session. When encountering a corresponding </script>, the filetype can be changed (eg back to plaintext).

4. Linear Classification Models

One way to express a linear model is by using the following linear equation:

where P is the total number of features, x _i is the i-th feature, β _i is the coefficient (weight) of the feature x _i , and C is the critical constant. In this example, C is the threshold for the verdict of maliciousness, and if the sum for a given file is less than C, the file is assigned a benign verdict, and if the sum is C or greater, the file is assigned a malicious verdict. means that

One approach for using the linear classification model by the data appliance 102 is as follows. A single float (d) is used to track the score of the incoming file, and a hash table is used to store the observed n-grams and corresponding coefficients (ie, x _i and β _i ). For each incoming packet, each of the n-gram characteristics is checked (eg, as provided by the security platform 122 ). Whenever a match is found for a feature (x _i ) in the hash table, a float (β _i ) matching that feature in the hash table is added (eg, to d ). When the end of the file is reached, a comparison of a single float d against a threshold C is performed to determine a decision for the file.

For n-gram counting, the feature (x _i ) is equal to the number of times the i-th n-gram is observed. Assume that the i-th n-gram is observed 4 times for a particular file. 4*β _i can be rewritten as β _i +β _i +β _i +β _i . Instead of counting how many times the i-th n-gram is observed (ie 4 times) and then multiplying by β _i , an alternative approach is to add β _i each time the ith n-gram is observed. Furthermore, suppose that the j-th n-gram is observed 3 times for the file. 3*β _j can be similarly written as β _j +β _j + β _j , counting how many times β _j is observed and then adding β _j instead of adding it at the end.

To find Σ(β _i x _i ), each of β _i x _i , β _j x _j , ... (where ... corresponds to all other features/weights) is added. This can be rewritten as β _i + β _i + β _i + β _j + β _j + β _j + β _j + .... Because additions are cumulative, additions of values are added in any order (eg, β _i + β _j + β _i + β _j + β _i + β _i + β _j + , etc.) and accumulate into a single float. Here, it is assumed that the float (d) starts at 0.0. Whenever feature x _i is observed, β _i can be added to float d, and whenever x _j is observed, β _j can be added to float d. This approach allows a 4 byte float to be used as a whole per session memory, in contrast to the approach where each session memory is proportional to the number of features, where the entire feature vector is stored in memory such that it can be multiplied by a weight vector . Using the example of 4 bytes * 1,000 4Kbyte features, 4K would be required for storage (compared to a single 4 byte float), which is 1,000 times more expensive.

5. Non-linear classification models

A variety of non-linear classification approaches may be used in conjunction with the techniques described herein. An example of a non-linear classification model is a gradient boosting tree. In this example, the feature vector is initialized to all-zero vectors. Unfortunately, for non-linear models (unlike previous models), the full set of features for which presence is detected (eg, 1,000 features) persists for the entire duration of the session. This is less efficient than the linear approach, but some efficiencies can still be obtained by down-sampling the features to 1 byte (0 to 255) rather than a full 4-byte float (as can be used on memory-unlimited devices).

As the data device 102 scans the file, each time a feature is observed, its value is incremented by one in the feature vector. Once the end of the file has been reached (or the end of feature observation has otherwise occurred), the constructed feature vector is fed into the gradient boosting tree model (eg, received from the secure platform 122 ). As will be described in more detail below, a non-linear classification model can be built using both n-grams (eg, 8-grams) and non-n-gram features. One example of a non-n-gram characteristic is the intended size of a file (which can be read as an out-of-packet value including the file's header). Any file data that appears after the intended end of the file (eg, as based on the file size specified in the header) is referred to as an overlay. Besides acting as a feature, the intended file length can be used as a proxy for how long the file is expected to be. A non-linear classifier hits the file's packet stream until the intended file length is reached, after which a verdict can be formed for the file regardless of whether the end of the file is actually reached. That a given file contains an overlay is also an example of a feature that can be used as part of a non-linear classification model. In various embodiments, the overlay portion of the file is not parsed, again - parsing may be performed before the actual end of the file. In other embodiments, feature extraction occurs and a malicious verdict is not formed until the actual end of the file is reached.

In an exemplary embodiment, the tree model includes 5,000 binary trees. Every node on each tree contains a feature and a corresponding threshold. An example of a portion of a tree is depicted in FIG. 5 . In the example shown in FIG. 5 , if the value for the feature (eg, feature F4) is less than a threshold (eg, 30), the left branch is taken 502 . If the value for the feature is above the threshold, the right branch is taken (504). The tree is searched until a leaf node is reached (eg, node 506), which has an associated value (eg, 0.7). The values of each leaf reached (for each of the trees) are summed (rather than multiplied) to obtain a final score to yield a decision. If the score is below the threshold, the file may be considered benign, if it is above the threshold, the file may be considered malicious. The lack of multiplication in obtaining the final score helps to make the use of the model more efficient in the resource constrained environment of the data device 102 .

In various embodiments, the trees themselves may be stored in a shared memory that is fixed on the data device 102 (until an updated model is received) and can be accessed by multiple sessions concurrently. The cost per session is the cost of storing the feature vector of the session, which can be manufactured out when the analysis of the session is completed.

6. Exemplary process

6 illustrates an example of a process for performing inline malware detection on a data device. In various embodiments, process 600 is performed by data appliance 102 , and in particular by threat engine 244 . The threat engine 244 may be implemented using a script (or set of scripts) written in a suitable scripting language (eg, Python). Process 600 may also be performed on an endpoint, such as client device 110 (eg, by an endpoint protection application running on client device 110 ).

Process 600 begins at 602 when an indication is received by device 102 that a file is being transmitted as part of a session. As an example of the processing performed at 602 , for a given session, the associated protocol decoder may invoke or otherwise use the appropriate file-specific decoder when the start of a file is detected by the protocol decoder. As described above, a filetype is determined (eg, by the decoder 402 ) and associated with a session (eg, until subsequent filetype analysis changes the filetype or file packets cease to be transmitted). so that it does not have to be done).

At 604 , n-gram analysis is performed on the sequence of received packets. As described above, the n-gram analysis may be performed inline with other analyzes being performed on the session by the device 102 . For example, while device 102 is performing analysis on a particular packet (eg, to check for the presence of a particular heuristic), it also ensures that any 8-grams in the packet are It can determine whether it matches the provided 8-grams. During processing performed at 604 , when an n-gram match is found, the corresponding pattern ID is used to map the condition to an action based on the filetype. The operation increments the weight counter (eg, when the filetype is associated with a linear classifier) or updates the feature vector to account for a match (eg, when the filetype is associated with a non-linear classifier).

Analysis of n-grams continues on a packet-by-packet basis until an end-of-file condition or checkpoint is reached. At point 606, the appropriate model is used to determine a verdict for the file (ie, comparing the final values obtained using the model against a maliciousness threshold). As mentioned above, models incorporate n-gram features and may also incorporate other features (eg, in the case of a non-linear classifier).

Finally, at 608 , an action is taken in response to the determination made at 606 . An example of a response action is to end the session. Another example of a response action is to allow the session to continue, but prevent the file from being sent (and place it in an isolated area instead). In various embodiments, device 102 is configured to share its verdicts (whether positive verdicts, malicious verdicts, or both) with secure platform 122 . When the security platform 122 completes its independent analysis of the file, it may use the verdict reported by the device 102 for a variety of purposes, including evaluating the performance of the model that formed the verdict. there is.

An exemplary threat signature for a sample is shown in FIG. 7B . In particular, for a sample with a SHA-256 hash of "4d73f42438fb5a857915219cdfa9cbb4ce3f771ffed93af81b0528931e4813f8", the first value in each pair corresponds to a feature, and the second value corresponds to a count. In the example shown in FIG. 7B , features comprising numbers (eg, feature “3905”) correspond to n-gram features, and features including “J” and numbers (eg, feature “J18”) are non Corresponds to n-gram features.

In an exemplary embodiment, the secure platform 122 is configured to target a specific false positive rate (eg, 0.001) when generating models for use by devices such as the data device 102 . Thus, in some cases (eg, one of all 1000 files), data device 102 incorrectly indicates that a benign file is malicious when performing an inline analysis using a model according to the techniques described herein. can decide In such a scenario, if the security platform 122 then determines that the file is in fact benign, it may then be whitelisted (eg, by another device) so that it is not flagged as malicious.

One approach for whitelisting is for the security platform 122 to instruct the device 102 to add the file to a whitelist stored on the device 102 . Another approach is that the security platform 122 instructs the whitelist system 154 for false positives and the whitelist system 154 returns devices such as device 102 to the latest false positives. It is instructing to keep it as positive information). As mentioned previously, one problem with devices such as device 102 is that they are resource limited. One approach to minimizing the resources used to maintain a whitelist on a device is to use a Least Recently Used (LRU) cache to maintain the whitelist. The whitelist may include file hashes, and may also be based on other factors, such as feature vectors or hashes of feature vectors.

VI. building models

Returning to the environment depicted in FIG. 1 , as previously described, the security platform 122 is configured to perform static and dynamic analysis on the samples it receives. The secure platform 122 may receive samples for analysis from a variety of sources. As previously mentioned, one exemplary type of sample source is a data device (eg, data devices 102, 136, and 148). Other sources (eg, other security device vendors, security researchers, etc.) one or more third-party providers of samples) may also be used, if applicable.As will be described in more detail below, the secure platform 122 uses the aggregate of samples it receives to build models. may be used (eg, may be used by the security device 102 in accordance with embodiments of the techniques described herein).

In various embodiments, the static analysis engine 306 is configured to perform feature extraction on samples it receives (eg, while also performing other static analysis functions as described above). An example process (eg, by secure platform 122 ) for performing feature extraction is depicted in FIG. 8A . Process 800 begins at 802 when static analysis of the sample begins. During feature extraction 804 , all 8-grams (or other applicable n-grams in embodiments where 8-grams are not used) are extracted from among the processed sample (eg, sample 130 in FIG. 3 ). is extracted In particular, a histogram of 8-grams is extracted (eg with a hash table) in the sample being analyzed, indicating the number of times a given 8-gram is observed in the processed sample. One advantage of extracting 8-grams during feature analysis by the static analysis engine 306 is that when using samples obtained from third parties (e.g., constructing models ) that potential privacy and contractual issues could be mitigated. The extracted histogram is stored at 806 .

In various embodiments, the static analysis engine 306 stores the extracted histogram (eg, represented using a hash table) for a given sample along with the histograms extracted from other samples in storage 142 (eg, stored in a Hadoop cluster). Data in Hadoop is compressed and when operations are performed on Hadoop data, the requested data is decompressed on the fly. An exemplary hash table (expressed in JSON) for a file is shown in FIG. 7A. Line 702 represents the SHA-256 hash of the file. Line 704 represents the UNIX time at which sample 130 arrives at secure platform 122 . Line 706 indicates the count of n-grams in the overlay section (eg, d00fbf4e088bc366':1 indicates that one instance of 'd00fb4e088bc366' was found in the overlay section). Line 708 represents a count of each of the 8-grams present in the file. Line 710 indicates that the file has an overlay. Line 712 indicates that the file type of the files is ".exe". Line 714 represents the UNIX time when secure platform 122 finishes processing sample 130 . Line 716 represents each count of non-eight-gram features that the file hits. Finally, line 718 indicates (eg, by security platform 122 ) that the file has been determined to be malicious.

In an exemplary embodiment, the set of 8-gram histograms stored in the Hadoop cluster grows by approximately 3 terabytes of 8-gram histogram data each day. Histograms will correspond to both malicious and benign samples (eg, will be labeled as such, based on the results of other static and dynamic analyzes performed by the security platform 122 as described above).

A histogram of 8-grams extracted from the sample being analyzed will be approximately 10% larger than the file itself, and a typical sample will have a histogram containing approximately 1 million different 8-grams. The total number of different possible 8-grams is 2 ⁶⁴ . As noted above, conversely, classification models (eg, as part of a subscription) transmitted by secure platform 122 to devices such as data appliance 102 may, in various embodiments, contain only a few thousand features. (eg, 1,000 features). One exemplary way to reduce a set of potentially 2 ⁶⁴ features to the 1,000 most important features for use in a model is to use mutual information technology. Other approaches may also be used where applicable (eg, a Chi-squared score). The four required parameters include the number of malignant samples with the given characteristics, the number of benign samples with the given characteristics, the total number of malignant samples, and the total number of benign samples. One advantage of mutual information is that it can be used efficiently for very large data sets. In Hadoop, the mutual information approach can be performed in a single pass by distributing the task across multiple mappers, each responsible for handling a particular feature (i.e., stored in the Hadoop cluster dataset for a given filetype). through all 8-gram histograms). Those features with the highest reciprocal information may be selected as the set of features that, if applicable, are most representative of maliciousness and/or most representative of benignness. The resulting 1,000 features can then be used to build models (eg, linear classification models and non-linear classification models), if applicable. For example, to build a linear classification model, model builder 152 (implemented using a set of open source tools and/or scripts written in a suitable language such as Python) (e.g., described in section VA4 above) ) the device 102 stores the top 1,000 features and applicable weights as a set of n-gram features to test.

In some embodiments, a non-linear classification model is also built by model builder 152 using the top 1,000 (or other desired number) features. In other embodiments, the non-linear classification model is constructed primarily using the highest order features (eg, 950), as well as other, non-n-gram features that may also be detected during packet-by-packet feature extraction and analysis (eg, 950). For example, 50 such features). Some examples of non-n-gram features that can be incorporated into a non-linear classification model are: (1) the size of the header, (2) the presence or absence of a checksum in the file, (3) the number of sections in the file. , (4) the intended length of the file (as indicated in the PE file's header), (5) whether the file contains overlay parts, and (6) whether the file requires the Windows EFI subsystem to run the PE. .

In some embodiments, rather than using mutual information to select the top 1,000 features, a larger set of features (over-generated set of features) is determined. As an example, the top 5,000 features may be initially selected using mutual information. The 5,000 sets then do not scale well to very large datasets (eg full Hadoop dataset), and more effective conventional feature selection techniques (eg, bagging (eg, bagging) on a reduced set (eg 5,000 features). bagging)). Conventional feature selection techniques can be used to select the final 1,000 features from a set of 5,000 features identified using mutual information.

Once the final 1,000 features have been selected, an exemplary way to construct a non-linear model is to use an open source tool such as scikit-learning or XGBoost. If applicable, parameter tuning may be performed, such as by using cross-validation.

An exemplary process for generating a model is depicted in FIG. 8B . In various embodiments, process 850 is performed by secure platform 122 . Process 850 begins at 852 when a set of extracted features (eg, including n-gram features) is received. One exemplary way in which a set of features may be received is by reading features stored as a result of process 800 . At 854 , a reduced set of features is determined from the features received at 852 . As described above, an exemplary way to determine a reduced set of features is by using mutual information. Other approaches (eg, a chi-square score) may also be used. Not only that, but also as described above, combinations of techniques may also be used in 852/854, such as selecting an initial set of features using mutual information and refining the initial set using bagging or another suitable technique. there is. Finally, as also described above, once features are selected (eg, at 854), appropriate models are built at 856 (eg, using open source or other tools, and, if applicable, performing parameter tuning) ). Models (eg, generated by model builder 152 using process 850 ) may be sent to data device 102 and other applicable recipients (eg, data devices 136 and 148 ). There is (eg, as part of a subscription service).

In various embodiments, model builder 152 generates models (eg, linear and non-linear classification models) on a daily (or other applicable) basis. By performing process 850 or otherwise periodically generating models, security platform 122 ensures that models used by devices, such as device 102 , represent the most current types of malware threats (eg, immoral individuals). ) can help ensure that the most recently deployed ones are detected.

Whenever the newly-generated model is determined to be better than the existing model (eg, as determined based on a set of quality evaluation metrics exceeding a threshold), the updated models are sent to data devices such as data device 102 . can be transmitted. In some cases, these updates adjust the weights assigned to features. These updates can be easily deployed and adopted (eg, as real-time updates) to devices. In other cases, these updates adjust the features themselves. These updates can become more complex to deploy, as they may require patches to components of the device, such as a decoder. One advantage of using overtraining during model generation is that the decoder can take into account whether or not certain features can be detected.

In various embodiments, devices are required to deploy updates to models as they are received (eg, by secure platform 122 ). In other embodiments, devices are allowed to selectively deploy updates (at least for a period of time). As an example, when a new model is received by the device 102 , both the old model and the new model may be run simultaneously on the device 102 for a period of time (eg, the old model is used in production and The new model reports actions to be taken without actually taking them). The administrator of the device may indicate whether an existing model or a new model should be used to process traffic on the device (eg, based on which model performs better). In various embodiments, device 102 provides telemetry indicating information (eg, false positive statistical information) such as which model(s) are running on device 102 and how effective the model(s) are. is provided back to the security platform 122 .

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the present invention. The disclosed embodiments are illustrative and not restrictive.

Claims (40)

In the system,
As a processor:
store a set comprising one or more sample classification models on the networking device;
perform n-gram analysis on the sequence of received packets associated with the received file, wherein performing n-gram analysis includes using at least one stored sample classification model;
determine whether the received file is malicious based at least in part on an n-gram analysis of the received sequence of packets, and in response to determining that the file is malicious, prevent propagation of the received file;
configured, the processor; and
and a memory coupled to the processor and configured to provide instructions to the processor. The system of claim 1 , wherein the processor is configured to perform the n-gram analysis at least in part by comparing n-grams of the received packet against a predetermined list of n-grams. The system of claim 2 , wherein the predetermined list of n-grams is generated using a plurality of previously collected malware samples. The system of claim 1 , wherein the processor is further configured to determine a filetype associated with the file. 5. The system of claim 4, wherein the processor is configured to select a linear classification model from the set of one or more sample classification models based on the determined filetype associated with the file. 6. The system of claim 5, wherein performing n-gram analysis comprises accumulating a set of weights corresponding to observed n-grams. 7. The system of claim 6, wherein the weights are accumulated to a single float value. 5. The system of claim 4, wherein the processor is configured to select, from the one or more sample classification models, a non-linear classification model based on the determined filetype associated with the file. 9. The system of claim 8, wherein the non-linear classification model includes n-gram features and non-n-gram features. The system of claim 9 , wherein the at least one non-n-gram characteristic is associated with a file size. The system of claim 9 , wherein the at least one non-n-gram characteristic is associated with the presence of an overlay. 9. The system of claim 8, wherein performing the n-gram analysis comprises updating a value for a feature in a feature vector whenever the feature is matched. The system of claim 1 , wherein using the at least one stored sample classification model comprises running a non-linear classifier on the packet stream until a desired file length is reached. 14. The system of claim 13, wherein the intended file length is not an actual file length and a verdict is determined before the actual end of the file is reached. The system of claim 1 , wherein the processor is further configured to receive at least one updated classification model. 2. The system of claim 1, wherein the n-gram analysis is performed inline with other packet analyzes as a single pass analysis of a traffic stream. The system of claim 1 , wherein the processor is further configured to use a set of whitelisted n-grams when performing the n-gram analysis. The system of claim 1 , wherein the processor is further configured to send a copy of the received file to a secure platform and perform the n-gram analysis while awaiting a decision from the secure platform. A method comprising:
storing a set comprising one or more sample classifications on a networking device;
performing n-gram analysis on a sequence of received packets associated with a received file, wherein performing n-gram analysis comprises using at least one stored sample classification model performing the steps; and
determining that the received file is malicious based at least in part on n-gram analysis of the received sequence of packets, and in response to determining that the file is malicious, preventing propagation of the received file; How to. A computer program product embodied in a tangible computer-readable storage medium and including computer instructions, the computer program product comprising:
The computer instructions are:
store a set comprising one or more sample classifications on the networking device;
perform n-gram analysis on the sequence of received packets associated with the received file, wherein performing n-gram analysis includes using at least one stored sample classification model;
determining that the received file is malicious based at least in part on n-gram analysis of the received sequence of packets, and in response to determining that the file is malicious, prevent propagation of the received file. , computer program products. In the system,
As a processor:
receive a set of features, including a plurality of n-grams, extracted from the set of files;
determine a reduced set of features comprising at least some of the plurality of n-grams;
to use the reduced set of features to create a model usable by data instruments to perform inline malware analysis.
configured, the processor; and
and a memory coupled to the processor and configured to provide instructions to the processor. The system of claim 1 , wherein the set of features includes features extracted from a set of known malicious files. The system of claim 1 , wherein the set of features comprises features extracted from a set of known benign files. The system of claim 1 , wherein the reduced set of features is determined using mutual information. The system of claim 1 , wherein the reduced set of features is determined using a Chi-squared score. The system of claim 1 , wherein the generated model comprises n-gram features. 7. The system of claim 6, wherein the generated model further comprises non-n-gram features. 8. The system of claim 7, wherein the at least one non-n-gram characteristic is associated with a file size. 8. The system of claim 7, wherein the at least one non-n-gram characteristic is associated with a header size. The system of claim 7 , wherein the at least one non-n-gram characteristic is associated with at least one of the presence or absence of a checksum in the file. 8. The system of claim 7, wherein the at least one non-n-gram characteristic is associated with a number of sections in the file. 8. The system of claim 7, wherein the at least one non-n-gram characteristic is associated with a desired length of the file. The system of claim 7 , wherein the at least one non-n-gram characteristic is associated with whether the file includes an overlay. The system of claim 1 , wherein the model is a linear model. The system of claim 1 , wherein the model is a non-linear model. The system of claim 1 , wherein the plurality of n-grams are extracted during static analysis of the set of files. The system of claim 1 , wherein the model is transmitted to a first data device. 18. The method of claim 17, wherein in response to a false positive result reported by a second data device, the processor is configured to generate an updated model and send the updated model to the first data device. , system. In the method,
receiving a set of features extracted from the set of files, the set of features comprising a plurality of n-grams;
determining a reduced set of features comprising at least some of the plurality of n-grams; and
using the reduced set of features to create a model usable by a data instrument to perform inline malware analysis. A computer program product embodied in a tangible computer-readable storage medium and including computer instructions, the computer program product comprising:
The computer instructions are:
receive a set of features, including a plurality of n-grams, extracted from the set of files;
determine a reduced set of features comprising at least some of the plurality of n-grams;
and use the reduced set of features to generate a model usable by a data instrument to perform inline malware analysis.

Images