WO2013119098A1

WO2013119098A1 - Verifying selected advertisements based upon placement parameters of advertisers

Info

Publication number: WO2013119098A1
Application number: PCT/MK2012/000001
Authority: WO
Inventors: Carlos Pacheco; Monica CHAWATHE; Dejan DESOVSKI
Original assignee: Carlos Pacheco; Chawathe Monica; Desovski Dejan
Priority date: 2012-02-09
Filing date: 2012-02-09
Publication date: 2013-08-15

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer-readable media, for identifying a plurality of advertisements that are each associated with at least one placement parameter. Receiving at least one advertisement request and selecting, for the at least one advertisement request, at least one advertisement based in part on the associated at least one placement parameter. Receiving at least one advertising assertion that includes at least one criterion based upon one or more placement parameters. The selection of the at least one advertisement is verified by applying the at least one advertising assertion to the placement data for the at least one advertisement and the associated at least one placement parameter.

Description

VERIFYING SELECTED ADVERTISEMENTS BASED UPON PLACEMENT

PARAMETERS OF ADVERTISERS

BACKGROUND

[0001] Ad servers can serve a large volume of advertisements within various resources. Advertisements can be placed based upon various settings. For example, an advertisement can be placed based upon a geographic location. In addition, advertisements can be placed based upon pricing data. For example, an

advertisement can have a maximum cost-per-click (CPC) value, such that placement of the advertisement should not result in a cost higher than the maximum CPC value. Advertisers trust that an ad server honors their advertising settings.

SUMMARY

[0002] In general, one aspect of the subject matter described in this specification can be embodied in methods for identifying a plurality of advertisements that are each associated with at least one placement parameter. Receiving at least one

advertisement request and selecting, for the at least one advertisement request, at least one advertisement based in part on the associated at least one placement parameter. Receiving at least one advertising assertion that includes at least one criterion based upon one or more placement parameters. The selection of the at least one advertisement is verified by applying the at least one advertising assertion to the placement data for the at least one advertisement and the associated at least one placement parameter. Other implementations of this aspect include corresponding systems, apparatuses, and computer-readable media configured to perform the actions of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Otherfeatures, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0004] Fig. 1 is an example environment in which a verification system verifies the placement of advertisements in accordance with an illustrative implementation.

[0005] Fig. 2 is a timing diagram showing operation of a time skew window in accordance with an illustrative implementation.

[0006] Fig. 3 is another timing diagram showing operation of a time skew window in accordance with an illustrative implementation.

[0007] Fig. 4 is another timing diagram showing operation of a time skew window in accordance with an illustrative implementation.

[0008] Fig. 5 is a flow diagram of a process for verifying selected advertisements in accordance with an illustrative implementation.

[0009] Fig. 6 is a flow diagram of a process for defining a time skew window and verifying the placement of an advertisement in accordance with an illustrative implementation.

[0010] Fig. 7 is a block diagram of a computer system in accordance with an illustrative implementation.

[0011] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0012] A verification system can be used to verify that a placed advertisement was properly placed based upon placement parameters. In one implementation, each advertisement that is placed is checked against the advertiser's criteria for placing the advertisement. Detected errors can be used to diagnose issues with the advertising server. Figure 1 is an example environment in which a verification system verifies the placement of advertisements in accordance with an illustrative implementation. The example environment 100 includes a network 106, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 106 can connect resources 102, advertisers 104, and an advertisement management system 110. The example environment 100 may include any suitable number of resources 102 and advertisers 104.

[0013] A resource 102 is any data that is accessible over the network 106. For example, resources include, but are not limited to, web pages, images, audio files, etc. A resource 102 may be identified by a resource address that is associated with the resource 102, such as a uniform resource locator (URL). A user (not shown) can request a resource through the network 106 using a user device. For example, a user can request a resource from a cell phone, television, computing device, etc., using a web browser.

[0014] Data representing the requested resource 02 can be provided to the user device for presentation by the user device. The data representing the resource 102 can include data specifying a portion of the resource or a portion of a user display (e.g., a presentation location in a pop-up window or in a slot of a web page) in which advertisements can be presented. These specified portions of the resource or user display are referred to as advertisement slots. For example, advertising slots can be within a search results page generated based upon keywords provided to a search engine by a user. When a resource 102 or search results are requested by a user device, the advertisement management system 1 10 receives a request for

advertisements to be provided with the resource 102 or search results. The request for advertisements can include characteristics of the advertisement slots that are defined for the requested resource or search results page and that can be provided to the advertisement management system 110. For example, a reference (e.g., URL) to the resource for which the advertisement slot is defined, a size of the advertisement slot, and/or media types that are available for presentation in the advertisement slot can be provided to the advertisement management system 10. Similarly, keywords (i.e., one or more words that are associated with content) associated with a requested resource ("resource keywords") or a search query for which search results are requested can also be provided to the advertisement management system 1 10 to facilitate

identification of advertisements that are relevant to the resource or search query.

[0015] Advertisers 104 can organize their advertisements into various groups. For example, advertisers can group advertisements together into a group. Advertisers 104 can also place one or more advertisements and/or one or more groups into an advertising campaign.

[0016] Based on data included in the request for advertisements, the advertisement management system 1 10 can select advertisements that are eligible to be provided in response to the request ("eligible advertisements"). Eligible advertisements can be selected based upon various placement parameters that are controlled by the advertiser. A placement parameter is a parameter that determines when an

advertisement can be placed in an advertising slot. Placement parameters provide advertisers with control on how their advertisements will be placed in resources. For example, placement parameters can include maximum daily budgets, keywords, time of day, day of week, maximum number of daily impressions, demographic criteria, maximum cost per click, type of devices to display ads on, start date, end date, etc. For example, an advertiser can indicate that a particular advertisement should only be shown at a particular time of day, day of the week, to certain types of devices, to users of particular demographics, to users that provided particular keywords, etc. As another example, an advertiser can specify that one or more advertisements should be shown from 5:00 pm through 12:00 am, Monday through Thursday on mobile devices. Changes to placement parameters may take time to propagate through the advertising management system 1 10.

[0017] Advertisers can provide various placement parameters for one or more of their advertisements at a variety of levels. For example, an advertiser can set placement parameters at the campaign level. As an example, a particular advertising campaign can have its own placement parameters, e.g., a maximum daily budget, a start/end date, a maximum CPC, a maximum number of impressions during a particular time period, previously selected advertisements from the advertiser, etc. As a specific example, an advertiser can specify for a campaign a maximum daily budget of $5.00 and advertisements in the campaign should not be displayed more than 100 times in a day. In addition, advertisers can have placement parameters associated with a particular advertisement. For example, a particular advertisement can have a maximum daily budget, and another advertisement can have a different maximum daily budget.

[0018] The advertisement management system 1 10 selects an eligible

advertisement for each advertisement slot (selected advertisements) of a resource 102. An eligible advertisement can be selected based upon comparing the advertisers' placement parameters with data associated with an advertisement slot, such as user data, and/or environment data. For example, data associated with a user can include keywords provided by the user and the type of device being used by the user.

Environment data can include time of day, day of the week, the number of times an advertisement was previously displayed, etc. Eligible advertisements can be selected such that the selection of the advertisement is consistent with an advertiser's

placement parameters that correspond to the advertisement. For example, in one implementation eligible advertisements can include the advertisements that have the highest bid amount that match the data associated with a particular advertising slot. As a particular example, a search results page can be generated based upon a user provided keyword and can include one or more advertising slots. Data associated with the user can indicate that the user is using a mobile device. In addition, environmental data can include the time of day that the search was instantiated. Eligible advertisements can be determined by finding advertisements whose placement parameters match the user data and/or environmental data. In one implementation, the advertisements with the highest bid amounts are selected as the eligible

advertisements.

[0019] The advertisement management system 1 10 can log the advertisements that were selected for a particular resource. For example, the advertisement management system 110 can store the data in one or more log files 1 12. The logs files 1 12 can be aggregated together. In some implementations, the log files 1 12 can be aggregated into a data store, such as a database.

[0020] A verification system 1 14 can compare the selection of an advertisement with the advertiser's placement parameters to verify that the selected advertisement is valid relative to the advertiser's placement parameters. In one implementation, the verification system 1 14 can verify each advertisement that is placed by the

advertisement management system 102. In this implementation, if the verification system 114 does not indicate any failures, the advertisement management system 110 properly selected every served advertisement. If an advertisement is not verified, this indicates that an advertisement was placed that was inconsistent with the advertiser's placement parameters.

[0021] To verify an advertisement, the verification system 1 4 can use one or more assertions to determine if the selection of the advertisement was correct. An assertion can be written to verify that the placement of an advertisement is consistent with the placement parameters associated with the advertisement. Assertions can include boolean logic, inequalities, string matching, pattern matching, etc. For example, an assertion can check to verify that a cost per click value associated with the placement of the advertisement is less than a maximum cost per click associated with the advertisement. The verification system 1 14 can include an application program interface (API) for assertions, such that assertions can be added to the verification system 114 without requiring changes to the verification system 1 14. Assertions can verify one or more aspects of the selection of advertisement. For example, an assertion could verify that the advertisement was properly selected for only a particular type of device or devices, e.g. mobile devices, tablets, desktops, laptops, etc. As another example, an assertion can verify that the CPC of a selected advertisement was below the advertiser's maximum CPC for the selected advertisement.

[0022] Advertisers can provide placement parameters based upon local values. For example, an advertiser can specify a maximum CPC in a currency local to the advertiser. The advertisement management system 110, however, may use a different currency when selecting advertisements. To account for this difference, an assertion can convert one or both of the CPCs such that they are in a common currency and then determine if the placed CPC is below the advertiser's maximum CPC.

[0023] The verification system 1 14 can test each of the assertions for each of the selected advertisements. In one implementation, data associated with each selected advertisement is logged. The assertions can use this logged data to verify that a placed advertisement was consistent with the advertiser's placement parameters associated with the selected advertisement. Any failed assertion can trigger one or more notification activities. For example, the verification system 1 14 can log that an error occurred. In another implementation, the verification system 1 14 can send an email, instant message, etc. to an administrator indicating that an assertion had failed along with the relevant information.

[0024] For example, in one embodiment, the verification system 1 14 can determine if an advertisement was placed in a manner that was inconsistent with the advertiser's placement parameters and resulted in an overcharge to the advertiser. For example, an advertisement that was placed and selected with a CPC of $2.50 would result in an overcharge if the advertiser's placement parameters included a maximum CPC of $1.50. In this example, the verification system 1 14 can determine that the advertiser was overcharged by $1.00. Based upon this determination, the verification system 1 14 can reimburse the advertiser for $1.00 to account for the overcharge. In one

implementation, the verification system 114 can log the advertiser and the amount of the overcharge, which can be reimbursed by another computing device, such as an accounting system.

[0025] Advertisers can change their placement parameters. For example, advertisers can change their placement parameters using a standard web interface. In one implementation, the web interface used to change placement parameters can be the same interface the advertiser uses to manage their advertisements. The

advertisement management system 1 10 can include multiple computing devices and changes to placement parameters can take time to propagate throughout the advertisement management system 1 10. For example, one computing device of the advertisement management system 110 can include placement parameters that are cached for performance reasons. These cached values can be updated periodically. The cached values, however, may not reflect the advertiser's latest changes.

Accordingly, an advertisement can be placed based upon outdated placement parameters that are inconsistent with updated placement parameters. According to an example embodiment, to avoid generating false positives, the verification system 1 14 can use a time skew arrangement to take into account the various values for the placement parameters in a window of time. In one implementation, a time skew can be a backwards looking time window that ends at the time an advertisement was served and goes back in time for a defined time skew duration. If tested placement

parameters are valid for any value of the advertising parameter within the time skew window, the tested advertising parameter can be deemed valid. For example, an assertion can test each of the advertising parameters within the time skew. Each of the multiple values of the placement parameter can be used in assertion to determine if the advertisement placement failed. In one implementation, if any of the parameter values passes, the advertisement placement is verified. In another implementation, the advertisement placement is verified only if all of the parameters are valid.

[0026] In another implementation, a time skew window is a forward looking time window that beings at a time a parameter (e.g., CPC) was changed and extends in time for the defined time skew duration. In this implementation, overlapping time skew windows represent multiple values of a placement parameter. To determine the values of a particular placement parameter, the time skew windows that include the time the advertisement was served are determined. The value of the placement parameter within each time skew window can be used in one or more assertions. Each of the multiple values of the placement parameter can be used in assertion to determine if the advertisement placement failed. In one implementation, if any of the parameter values passes, the advertisement placement is verified. In another implementation, the advertisement placement is verified only if all of the parameters are valid.

[0027] Figure 2 is a timing diagram showing operation of a time skew window in accordance with an illustrative implementation. At some point in time t=0, an advertiser can change their maximum CPC value to $10 (202). At a future point, t=100, the advertiser can change their maximum CPC value to $1 (204). An advertisement that has a CPC of $9 can be placed immediately after the CPC change at t=101. If the change in the advertiser's CPC has not propagated to the advertisement management system 110, placing the advertisement with a $9 CPC would be below the minimum CPC of $10. If, however, the verification system 114 accesses the advertiser's changed CPC value of $1 , the verification system 1 14 would indicate that the

advertisement was not properly placed in regard to the advertiser's parameters. To resolve this issue, the verification system 114 may use a time skew window 208. The verification system 1 14 determines all values of a parameter within the time skew window 208. The duration of the time skew window can be set to an observed amount of time a change to a parameter takes to propagate through the advertisement management system 1 10. For example, if it is empirically observed that it generally takes twelve hours for changes in placement parameters to completely propagate through the system, then the time skew window may be set at twelve hours. In another implementation, the length of the time skew window 208 can be set by an assertion. One time skew window may be used for each placement and the time skew window may extend an amount of time immediately preceding the moment at which an advertisement was placed. Returning to Figure 2, the maximum CPC values within the time skew window 208 are 10 and 1. Accordingly, an assertion would validate the CPC value of $9 at time t=101. [0028] In one implementation, the CPC values within the time skew window 208 can be determined by finding any changes to the CPC value within the time skew window 208 and also the last parameter change that occurred prior to the start of the time skew window 208. In the example shown in Figure 2, the last parameter change that occurred prior to the start of the time skew window 208 is at t=0. The last parameter change specifies the CPC value that is in effect during the first part of the time skew window 208 prior to any changes that occur within the time skew window 208 (e.g., during the time period 90≤ t <100). Accordingly, all of the CPC values within the time skew window 208 can be determined.

[0029] Figure 3 is another timing diagram showing operation of a time skew window in accordance with an illustrative implementation. A time skew window 308 looks backward from the advertisement placement that occurred at time t=101. In Figure 3 as in Figure 2, the placement is validated because there is at least one value within time skew window 308 for which the placement is valid. Specifically at some point in time, an advertiser sets their maximum CPC to $1 (302). The advertiser at some other point in time sets the maximum CPC to $10 (304) and then to $3 (306). At time t=100, an advertisement of the advertiser is placed with a CPC of $9. The values of the advertiser's CPC in a time skew window 308 are $1 , $10, and $3. An assertion would validate this placement since the $9 placed CPC is less than the $10 maximum CPC that was in effect from t=50 through t=74.

[0030] Figure 4 is another timing diagram showing operation of a time skew window in accordance with an illustrative implementation. A time skew window 408 looks backward from the advertisement placement that occurred at time t= 01. In Figure 4, the placement is not validated because if fails for all possible values within time skew window 408. Specifically at some initial point in time, t=0, the maximum CPC for an advertiser is set to $10. At a later time, t=10, this parameter is set to $1. The maximum CPC is changed again to $2 at t=50 and then to $3 at t=75. An

advertisement for the advertiser is placed at t= 00 with a CPC of $9. The values of a maximum CPC parameter within a time skew window are $1 , $2, and $3. Accordingly, an assertion comparing the placed CPC of $9 to each of the maximum CPC values within the time skew window 408 would fail. The maximum CPC value of $10 that was valid between t=0 and t=9 would not be considered by the assertion since it is outside of the time skew window 408.

[0031] In one implementation, an overcharge can be determined based upon the maximum CPC values. For example, an overcharge of $6 can be calculated based upon the maximum CPC value of $3 within the time skew window subtracted from the placed CPC amount of $9.

[0032] Figure 5 is a flow diagram of a process for verifying selected advertisements in accordance with an illustrative implementation. The process 500 can be

implemented on a computing device. In one implementation, the process 500 is encoded on a computer-readable medium that contains instructions that, when executed by a computing device, cause the computing device to perform operations of process 500.

[0033] Several advertisements, each associated with placement parameters that include a maximum CPC, can be received (502). For example, an advertisement management system 1 10 can receive the advertisements from one or more advertisers 104. A request for one or more advertisements can be received (504). For example, users can request a resource 102 that includes at least one advertising slot. The advertisement management system 1 10 can receive a request to select

advertisements for each of the advertising slots. In response to the advertising requests, an advertisement can be selected based on the placement parameters of the advertisements (506). For example, the advertising management system 10 can select the advertisements. In one implementation, an indication that each

advertisement was selected can be logged by the advertisement management system 1 10. In one implementation, the logged data can include all of the data corresponding to one or more placement parameters. For example, the data can include an advertisement identifier, a cost-per-click, a placement position, any keywords used to trigger the advertisement, a location identifier, etc. Multiple advertising assertions can be received (510). In one implementation, the verification system 1 14 can include or access the assertions. The verification system 1 14 can receive the advertising assertions. For example, a list of assertions can be read to determine what assertions are to be ran. As another example, the assertions that are located in one or more directories can be used as the list of assertions that are to be ran to verify the placement of advertisers. As described above, an assertion determines if data values associated with a selected advertisement are valid based upon an advertiser's placement parameters. Each assertion verifies the selection of each selected advertisement based upon the placement parameters as described above (512). For example, the assertions can compare the logged data to an advertiser's placement criteria using a time skew window, as described above. In one implementation, the verification system 1 14 executes each of the advertising assertions.

[0034] Figure 6 is a flow diagram of a process 600 for defining a time skew window and verifying the placement of an advertisement in accordance with an illustrative implementation. The process 600 can be implemented on a computing device. In one implementation, the process 600 is encoded on a computer-readable medium that contains instructions that, when executed by a computing device, cause the computing device to perform operations of process 600.

[0035] An advertisement can be placed in a resource 602. For example, the advertising management system 110 can select an advertisement for a search results page. At some later point in time, the placement of the advertisement can be verified against the advertiser's placement parameters. Placement parameters are not static and may change based upon input from the advertiser. To account for changes in placement parameters, a time skew window can be used. In one implementation, a time skew window is defined based upon the placement of an advertisement 604. For example, the time skew window end at the placement of an advertisement and be a predetermine width. Accordingly, the time skew window looks back in time from the placement of the advertisement. The values for placement parameters within this window are identified 606. For example, the verification system 1 14 can query the advertising management system 1 10 for all values of a placement parameter within the time skew window. Thus, a particular placement parameter can have one or more values. The selection of the advertisement can be verified using the one or more placement parameters 608. For example, the verification system 114 can verify that the advertiser's placement parameters are consistent with the parameters that were used to place the advertisement. In one implementation, a particular placement parameter is verified if at least one value of the parameter is consistent with the corresponding parameter used to place the advertisement. In another implementation, a particular placement is verified if each parameter value is consistent with the corresponding parameter used to place the advertisement.

[0036] Figure 7 is a block diagram of a computer system in accordance with an illustrative implementation. The computer system or computing device 700 can be used to implement a client, servers, networking components, cloud computing resources, etc. The computing system 700 includes a bus 705 or other communication component for communicating information and a processor 710 or processing circuit coupled to the bus 705 for processing information. The computing system 700 can also include one or more processors 710 or processing circuits coupled to the bus for processing information. The computing system 700 also includes main memory 715, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 705 for storing information, and instructions to be executed by the processor 710. Main memory 715 can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor 710. The computing system 700 may further include a read only memory (ROM) 710 or other static storage device coupled to the bus 705 for storing static information and instructions for the processor 710. A storage device 725, such as a solid state device, magnetic disk or optical disk, is coupled to the bus 705 for persistently storing information and instructions.

[0037] The computing system 700 may be coupled via the bus 705 to a display 735, such as a liquid crystal display, or active matrix display, for displaying information to a user. An input device 730, such as a keyboard including alphanumeric and other keys, may be coupled to the bus 705 for communicating information and command

selections to the processor 710. In another implementation, the input device 730 has a touch screen display 735. The input device 730 can include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 710 and for controlling cursor movement on the display 735.

[0038] According to various implementations, the processes described herein can be implemented by the computing system 700 in response to the processor 710 executing an arrangement of instructions contained in main memory 715. Such instructions can be read into main memory 715 from another computer-readable medium, such as the storage device 725. Execution of the arrangement of instructions contained in main memory 715 causes the computing system 700 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained in main memory 715. In alternative implementations, hard-wired circuitry may be used in place of or in combination with software instructions to effect illustrative implementations. Thus, implementations are not limited to any specific combination of hardware circuitry and software.

[0039] Although an example computing system has been described in Figure 7, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0040] Implementations of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on one or more non-transitory computer storage media for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a non-transitory computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). Accordingly, the computer storage medium is both tangible and non-transitory.

[0041] The operations described in this specification can be performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0042] The term "data processing apparatus" or "computing device" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC

(application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. [0043] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing

environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a jDortion of a file that holds other programs or data (e.g., one or more scripts stored in. a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0044] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0045] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0046] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain features described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0047] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0048] Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

WHAT IS CLAIMED IS:

1. A method comprising:

identifying a plurality of advertisements, wherein each advertisement is associated with at least one placement parameter;

receiving at least one advertisement request;

selecting, for the at least one advertisement request, at least one advertisement based in part on the associated at least one placement parameter;

receiving at least one advertising assertion, wherein each advertising assertion comprises at least one criterion based upon one or more placement parameters; and

verifying the selection of the at least one advertisement by applying the at least one advertising assertion to the placement data for the at least one advertisement and the associated at least one placement parameter.

2. The method of claim 1 , wherein the at least one placement parameter comprise all distinct values of the at least one placement parameter within a predetermined time window, wherein the value of the at least one placement parameter can change over time.

3. The method of claim 2, wherein verifying the selection of the at least one advertisement comprises the at least one advertising assertion verifying at least one of the distinct values of the placement parameter.

4. The method of claim 2, wherein verifying the selection of the at least one advertisement comprises the at least one advertising assertion verifying each of the distinct values of the placement parameter.

5. The method of claim 1 , further comprising receiving a new value for a placement parameter associated with one of the plurality of advertisements.

6. The method of claim 1 , wherein the at least one placement parameter comprises a maximum cost-per-click, and further comprising receiving a new maximum cost-per-click value for one of the plurality of advertisements.

7. The method of claim 6, wherein the at least one advertising assertion comprises comparing the maximum cost-per-click for an advertisement with the cost-per-click from the placement data.

8. The method of claim 7, wherein verifying the at least one advertising assertion comprises comparing the cost-per-click from the placement data with both the original maximum cost-per-click value and the new maximum cost-per- click value.

9. The method of claim 1 , wherein one or more advertisements are associated with an advertising campaign, wherein the advertising campaign comprises campaign parameters, and wherein selecting one or more advertisements is based upon the campaign parameters.

10. The method of claim 9, wherein the at least one advertising assertion is based upon one or more campaign parameters.

11. The method of claim 9, wherein the placement parameters for an

advertisement comprise campaign parameters from the associated advertising campaign.

12. The method of claim 1 , wherein the at least one placement parameter comprises a start date, an end date, a list of keywords, a device parameter, a budget amount, a maximum number of impressions, a language, and a location.

13. The method of claim 12, wherein the location comprises a country or a city.

14. The method of claim 1 , further comprising:

determining whether the at least one assertion has failed; and providing a notification to the advertiser based upon determining the at least one assertion has failed.

15. The method of claim 1 , further comprising:

determining an amount of a refund based upon a failed assertion; and crediting an account of an advertiser for the amount of the refund.

16. A method comprising:

Identifying a plurality of advertisements, wherein each advertisement is associated with at least one placement parameter, wherein one or more advertisements are associated with an advertising campaign, and wherein the advertising campaign comprises at least one campaign parameter;

receiving at least one advertisement request;

selecting, for the at least one advertisement request, at least one advertisement based in part on the associated at least one placement parameter or the associated at least one campaign parameter;

receiving at least one advertising assertion, wherein each advertising assertion comprises at least one criterion based upon a placement parameter or a campaign parameter; and

verifying the selection of the at least one advertisement by applying the at least one advertising assertion to the placement data for the at least one advertisement and the associated at least one placement parameter or the associated at least one campaign parameter, wherein the associated at least one placement parameter or the associated at least one campaign parameter comprise all distinct values of the parameter within a predetermined time.

17. A non-transitory computer-readable medium having instructions stored thereon, the instructions comprising: instructions to identify a plurality of advertisements, wherein each advertisement is associated with at least one placement parameter;

instructions to receive at least one advertisement request;

instructions to select, for the at least one advertisement request, at least one advertisement based in part on the associated at least one placement parameter;

instructions to receive at least one advertising assertion, wherein the at least one advertising assertion comprises at least one criterion based upon one or more placement parameters; and

instructions to verify, the selection of the at least one advertisement by applying the at least one advertising assertion to the to the placement data for the at least one advertisement and the associated at least one placement parameter.

18. The non-transitory computer-readable medium of claim 17, further comprising receiving a new value for a placement parameter associated with one of the plurality of advertisements.

19. The non-transitory computer-readable medium of claim 17, wherein the at least one advertising assertion comprises comparing a maximum cost-per-click for an advertisement with a cost-per-click from the placement data.

20. The non-transitory computer-readable medium of claim 17, further comprising:

instructions to determine an amount of a refund based upon a failed assertion; and

instructions to credit an account of an advertiser for the amount of the refund.