short-paper

When web personalization misleads bucket testing

Authors:

Ariyam Das,

Harish RanganathAuthors Info & Claims

UEO '13: Proceedings of the 1st workshop on User engagement optimization

Pages 17 - 20

https://doi.org/10.1145/2512875.2512879

Published: 01 November 2013 Publication History

Get Access

Abstract

Online service platforms like search engines, news websites, information portal sites and others offer highly personalized content to the users based on their interest and taste. At the same time, these online sites, with large audiences, frequently use bucket testing to evaluate the impact of a new feature or service on a small subset of its users before releasing it to the entire user population. In general, web personalization leads to an improved user engagement for the sites, but it can also interfere and adversely impact the online bucket testing experiments. In this work, we show empirically through real experiments conducted on Yahoo pages that how personalization can mislead to erroneous interpretation of the bucket testing results. We also present a novel algorithmic framework that addresses this challenge and draws a more accurate inference from the bucket testing results by factoring in the personalization experience of the users. The effectiveness of our algorithm is demonstrated through experiments conducted on Yahoo pages.

References

[1]

Maurice D. Mulvenna, Sarabjot S. Anand and Alex G. Büchner. "Personalization on the Net using Web mining: introduction." Communications of the ACM 43.8 (2000): 122--125.

Digital Library

Google Scholar

[2]

W. Kim. "Personalization: Definition, Status, and Challenges Ahead." In Proceedings of Journal of Object Technology 2002, 29--40.

Google Scholar

[3]

Lars Backstrom and Jon Kleinberg. "Network bucket testing." In Proceedings of WWW 2011, pp. 615--624.

Digital Library

Google Scholar

[4]

Anirban Majumder and Nisheeth Shrivastava. "Know Your Personalization: Learning Topic Level Personalization in Online Services." In Proceedings of WWW 2013, pp. 873--884.

Digital Library

Google Scholar

[5]

Bamshad Mobasher, Robert Cooley and Jaideep Srivastava. "Automatic personalization based on Web usage mining." Communications of the ACM 43.8 (2000): 142--151.

Digital Library

Google Scholar

[6]

Panagiotis Germanakos and Constantinos Mourlas. "Adaptation and personalization of web-based multimedia content." Multimedia transcoding in mobile and wireless networks (2008): 160--177.

Google Scholar

[7]

Liran Katzir, Edo Liberty and Oren Somekh. "Framework and algorithms for network bucket testing." In Proceedings of WWW 2012, pp. 1029--1036.

Digital Library

Google Scholar

[8]

http://cdn.ttgtmedia.com/rms/SearchBusinessIntelligence_IN/IE_2012-SS-Feb-Feb-bucket%20testing%20framework- Yahoo_Ashish%20Vikram.pdf

Google Scholar

[9]

Ariyam Das and Harish Siddapura Ranganath. "Effective Interpretation of Bucket Testing Results through Big Data Analytics." IEEE International Congress on BigData, 2013.

Digital Library

Google Scholar

Cited By

View all

Esteller-Cucala MFernandez VVilluendas D(2020)Evaluating Personalization: The AB Testing Pitfalls Companies Might Not Be Aware of—A Spotlight on the Automotive Sector WebsitesFrontiers in Artificial Intelligence10.3389/frai.2020.000203Online publication date: 9-Apr-2020
https://doi.org/10.3389/frai.2020.00020
Liu JLiu CBelkin N(2020)Personalization in text information retrievalJournal of the Association for Information Science and Technology10.1002/asi.2423471:3(349-369)Online publication date: 28-Jan-2020
https://dl.acm.org/doi/10.1002/asi.24234
Esteller-Cucala MFernandez VVilluendas DPapadopoulos GSamaras GWeibelzahl SJannach DSantos O(2019)Experimentation Pitfalls to Avoid in A/B Testing for Online PersonalizationAdjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization10.1145/3314183.3323853(153-159)Online publication date: 6-Jun-2019
https://dl.acm.org/doi/10.1145/3314183.3323853
Show More Cited By

Index Terms

When web personalization misleads bucket testing
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Generating semantically enriched user profiles for Web personalization

Traditional collaborative filtering generates recommendations for the active user based solely on ratings of items by other users. However, most businesses today have item ontologies that provide a useful source of content descriptors that can be used ...
Personalized Itinerary Recommendation with Queuing Time Awareness
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Personalized itinerary recommendation is a complex and time-consuming problem, due to the need to recommend popular attractions that are aligned to the interest preferences of a tourist, and to plan these attraction visits as an itinerary that has to be ...
When does web-based personalization really work? The distinction between actual personalization and perceived personalization

A number of prior studies have suggested that personalization is more efficacious than non-personalization. However, the existing literature is somewhat ambiguous on whether the test of personalization effects should be based on a message sender's ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

UEO '13: Proceedings of the 1st workshop on User engagement optimization

November 2013

36 pages

ISBN:9781450324212

DOI:10.1145/2512875

Program Chairs:
Liangjie Hong
Yahoo Labs, USA
,
Shuang-Hong Yang
Twitter Inc., USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

CIKM'13

Sponsor:

CIKM'13: 22nd ACM International Conference on Information and Knowledge Management

November 1, 2013

California, San Francisco, USA

Acceptance Rates

UEO '13 Paper Acceptance Rate 6 of 6 submissions, 100%;

Overall Acceptance Rate 6 of 6 submissions, 100%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
149
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Esteller-Cucala MFernandez VVilluendas D(2020)Evaluating Personalization: The AB Testing Pitfalls Companies Might Not Be Aware of—A Spotlight on the Automotive Sector WebsitesFrontiers in Artificial Intelligence10.3389/frai.2020.000203Online publication date: 9-Apr-2020
https://doi.org/10.3389/frai.2020.00020
Liu JLiu CBelkin N(2020)Personalization in text information retrievalJournal of the Association for Information Science and Technology10.1002/asi.2423471:3(349-369)Online publication date: 28-Jan-2020
https://dl.acm.org/doi/10.1002/asi.24234
Esteller-Cucala MFernandez VVilluendas DPapadopoulos GSamaras GWeibelzahl SJannach DSantos O(2019)Experimentation Pitfalls to Avoid in A/B Testing for Online PersonalizationAdjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization10.1145/3314183.3323853(153-159)Online publication date: 6-Jun-2019
https://dl.acm.org/doi/10.1145/3314183.3323853
Conti MGangwal AGochhayat STolomei G(2018)Spot the Difference: Your Bucket is Leaking : A Novel Methodology to Expose A/B Testing Effortlessly2018 IEEE Conference on Communications and Network Security (CNS)10.1109/CNS.2018.8433122(1-7)Online publication date: May-2018
https://doi.org/10.1109/CNS.2018.8433122

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Generating semantically enriched user profiles for Web personalization

Personalized Itinerary Recommendation with Queuing Time Awareness

When does web-based personalization really work? The distinction between actual personalization and perceived personalization