Article

Lessons learned from using an online platform to conduct large-scale, online controlled security experiments with software developers

Authors:

Sascha FahlAuthors Info & Claims

CSET'17: Proceedings of the 10th USENIX Conference on Cyber Security Experimentation and Test

Page 6

Published: 14 August 2017 Publication History

Publisher Site

Abstract

Security and privacy researchers are increasingly conducting controlled experiments focusing on IT professionals, such as software developers and system administrators. These professionals are typically more difficult to recruit than general end-users. In order to allow for distributed recruitment of IT professionals for security user studies, we designed Developer Observatory, a browser-based virtual laboratory platform that enables controlled programming experiments while retaining most of the observational power of lab studies. The Developer Observatory can be used to conduct largescale, reliable online programming studies with reasonable external validity. We report on our experiences and lessons learned from two controlled programming experiments (n>200) conducted using Developer Observatory.

References

[1]

ACAR, Y., BACKES, M., FAHL, S., GARFINKEL, S., KIM, D., MAZUREK, M. L., AND STRANSKY, C. Comparing the Usability of Cryptographic APIs. In Proc. 38th IEEE Symposium on Security and Privacy (SP'17) (2017), IEEE.

Google Scholar

[2]

ACAR, Y., BACKES, M., FAHL, S., KIM, D., MAZUREK, M. L., AND STRANSKY, C. You Get Where You're Looking For: The Impact of Information Sources on Code Security. In Proc. 37th IEEE Symposium on Security and Privacy (SP'16) (2016), IEEE.

Google Scholar

[3]

ACAR, Y., FAHL, S., AND MAZUREK, M. L. You are Not Your Developer, Either: A Research Agenda for Usable Security and Privacy Research Beyond End Users. In Proc. IEEE Secure Development Conference (SecDev'16) (2016), IEEE.

Google Scholar

[4]

ACAR, Y., STRANSKY, C., WERMKE, D., MAZUREK, M. L., AND FAHL, S. Security Developer Studies with GitHub Users: Exploring a Convenience Sample. In Proc. 13th Symposium on Usable Privacy and Security (SOUPS'17) (2017), USENIX Association.

Google Scholar

[5]

BENZEL, T. The science of cyber security experimentation: the DETER project. In Proc. 27th Annual Computer Security Applications Conference (ACSAC'11) (2011), ACM.

Crossref

Google Scholar

[6]

DI PENTA, M., STIREWAL, R., AND KRAEMER, E. Designing your Next Empirical Study on Program Comprehension. In Proc. 15th IEEE International Conference on Program Comprehension (ICPC'07) (2007), IEEE.

Crossref

Google Scholar

[7]

FORGET, A., KOMANDURI, S., ACQUISTI, A., CHRISTIN, N., CRANOR, L. F., AND TELANG, R. Security behavior observatory: Infrastructure for long-term monitoring of client machines. Tech. rep., Carnegie Mellon University, CyLab, 2014.

Google Scholar

[8]

LVESQUE, F. L., AND FERNANDEZ, J. M. Computer security clinical trials: Lessons learned from a 4-month pilot study. In Proc. 7th USENIX Workshop on Cyber Security Experimentation and Test (CSET'14) (2014), USENIX Association.

Crossref

Google Scholar

[9]

NANDUGUDI, A., MAITI, A., KI, T., BULUT, F., DEMIRBAS, M., KOSAR, T., QIAO, C., KO, S. Y., AND CHALLEN, G. PhoneLab: A Large Programmable Smartphone Testbed. In Proc. 1st International Workshop on Sensing and Big Data Mining (SENSEMINE'13) (2013), ACM.

Crossref

Google Scholar

[10]

RUEF, A., HICKS, M., PARKER, J., LEVIN, D., MAZUREK, M. L., AND MARDZIEL, P. Build It, Break It, Fix It: Contesting Secure Development. In Proc. 23nd ACM Conference on Computer and Communication Security (CCS'16) (2016), ACM.

Crossref

Google Scholar

[11]

SIATERLIS, C., AND MASERA, M. A Review of Available Software for the Creation of Testbeds for Internet Security Research. In Proc. 1st International Conference on Advances in System Simulation (SIMUL'09) (2009), IEEE.

Crossref

Google Scholar

[12]

SIEGMUND, J., SIEGMUND, N., AND APEL, S. Views on Internal and External Validity in Empirical Software Engineering. In Proc. 37th IEEE International Conference on Software Engineering (ICSE'15) (2015), IEEE.

Crossref

Google Scholar

[13]

WASH, R., RADER, E., AND FENNELL, C. Can People Self-Report Security Accurately?: Agreement Between Self-Report and Behavioral Measures. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI'17) (2017), ACM.

Crossref

Google Scholar

Cited By

View all

Serafini RHorstmann SNaiakshina ABalzarotti DXu W(2024)Engaging company developers in security research studiesProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699084(3277-3294)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699084
Serafini RGutfleisch MHorstmann SNaiakshina AKelley PKapadia A(2023)On the recruitment of company developers for security studies: results from a qualitative interview studyProceedings of the Nineteenth USENIX Conference on Usable Privacy and Security10.5555/3632186.3632204(321-340)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.5555/3632186.3632204
Geierhaas LOrtloff ASmith MNaiakshina AChiasson SKapadia A(2022)Let's hashProceedings of the Eighteenth USENIX Conference on Usable Privacy and Security10.5555/3563609.3563636(503-522)Online publication date: 8-Aug-2022
https://dl.acm.org/doi/10.5555/3563609.3563636
Show More Cited By

Recommendations

Online controlled experiments: introduction, learnings, and humbling statistics
RecSys '12: Proceedings of the sixth ACM conference on Recommender systems

The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). Whether for front-end user-interface changes, or backend ...
Online controlled experiments: introduction, insights, scaling, and humbling statistics
UEO '13: Proceedings of the 1st workshop on User engagement optimization

The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). From front-end user-interface changes to backend algorithms, ...
Online controlled experiments at large scale
KDD '13: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining

Web-facing companies, including Amazon, eBay, Etsy, Facebook, Google, Groupon, Intuit, LinkedIn, Microsoft, Netflix, Shop Direct, StumbleUpon, Yahoo, and Zynga use online controlled experiments to guide product development and accelerate innovation. At ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CSET'17: Proceedings of the 10th USENIX Conference on Cyber Security Experimentation and Test

August 2017

10 pages

Editors:
José M. Fernandez
École Polytechnique de Montréal
,
Mathias Payer
Purdue University

Publisher

USENIX Association

United States

Publication History

Published: 14 August 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Serafini RHorstmann SNaiakshina ABalzarotti DXu W(2024)Engaging company developers in security research studiesProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699084(3277-3294)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699084
Serafini RGutfleisch MHorstmann SNaiakshina AKelley PKapadia A(2023)On the recruitment of company developers for security studies: results from a qualitative interview studyProceedings of the Nineteenth USENIX Conference on Usable Privacy and Security10.5555/3632186.3632204(321-340)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.5555/3632186.3632204
Geierhaas LOrtloff ASmith MNaiakshina AChiasson SKapadia A(2022)Let's hashProceedings of the Eighteenth USENIX Conference on Usable Privacy and Security10.5555/3563609.3563636(503-522)Online publication date: 8-Aug-2022
https://dl.acm.org/doi/10.5555/3563609.3563636
Fischer FStachelscheid YGrossklags JKim YKim JVigna GShi E(2021)The Effect of Google Search on Software Security: Unobtrusive Security Interventions via Content Re-rankingProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security10.1145/3460120.3484763(3070-3084)Online publication date: 12-Nov-2021
https://dl.acm.org/doi/10.1145/3460120.3484763
Danilova ANaiakshina ADeuter JSmith MLipford H(2020)Replication: on the ecological validity of online security developer studiesProceedings of the Sixteenth USENIX Conference on Usable Privacy and Security10.5555/3488905.3488915(165-183)Online publication date: 10-Aug-2020
https://dl.acm.org/doi/10.5555/3488905.3488915
Ukrop MKraus LMatyas V(2020)Will You Trust This TLS Certificate?Digital Threats: Research and Practice10.1145/34194721:4(1-29)Online publication date: 10-Dec-2020
https://dl.acm.org/doi/10.1145/3419472
Assal HChiasson S(2018)Security in the software development lifecycleProceedings of the Fourteenth USENIX Conference on Usable Privacy and Security10.5555/3291228.3291251(281-296)Online publication date: 12-Aug-2018
https://dl.acm.org/doi/10.5555/3291228.3291251
Gorski PIacono LWermke DStransky CMoeller SAcar YFahl S(2018)Developers deserve security warnings, tooProceedings of the Fourteenth USENIX Conference on Usable Privacy and Security10.5555/3291228.3291250(265-280)Online publication date: 12-Aug-2018
https://dl.acm.org/doi/10.5555/3291228.3291250
Acar YStransky CWermke DMazurek MFahl S(2017)Security developer studies with github usersProceedings of the Thirteenth USENIX Conference on Usable Privacy and Security10.5555/3235924.3235932(81-95)Online publication date: 12-Jul-2017
https://dl.acm.org/doi/10.5555/3235924.3235932

Abstract

References

Cited By

Recommendations

Online controlled experiments: introduction, learnings, and humbling statistics

Online controlled experiments: introduction, insights, scaling, and humbling statistics

Online controlled experiments at large scale

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations