A Systematic Way to LTE Testing
Article No.: 7, Pages 1 - 15
Abstract
LTE test cases are standardized by 3GPP. They must be executed on every LTE-capable device model before commercial release. In this work, we examine the LTE testing practices in terms of completeness and efficiency. We discover that the standardized tests are incomplete in that a number of test cases related to multiple protocol interactions are missing. Our analysis also shows that, the isolated treatment of test cases, but not from the system perspective, incurs repetitive executions of test operations, thus resulting in testing inefficiencies. We thus make a case for a paradigm shift from ad hoc testing to a methodical approach to LTE testing. We follow a few guidelines from the LTE standards and propose an algorithmic approach to systematic testing. In the process, we address various challenges, provide complete list of test cases, and present the related algorithms. Our evaluation shows that, by eliminating repetitive operations, our new scheme reduces up to 70% of LTE testing steps. We also find 87 new, yet valid test cases that are not defined by the LTE standards.
References
[1]
3GPP TS 36.523--1: Protocol conformance specification, 2018.
[2]
Anite maintains LTE conformance testing lead. http://www.anite.com/businesses/handset-testing/n ews/anite-maintains-lte-conformance-testing-lead.
[3]
Anritsu conformance test systems. https://www.anritsu.com/en-US/test-measurement/m obile-wireless-communications/conformance-test-s ystems.
[4]
Anite conformance test systems. http://www.anite.com/businesses/handset-testing/our -products.
[5]
Anritsu: How conformance tests are carried out. http://dl.cdn-anritsu.com/en-en/test-measurement/fil es/Product-Introductions/Product-Introduction/me78 73la-el1100.pdf.
[6]
Anite test documents. http://www.keysight.com/en/pd-2372474-pn-E7515A/ uxm-wireless-test-set?pm=PL&nid=-33762.1078013 &cc=US&lc=eng.
[7]
UE demonstration of conformance testing -- Anite. http://www.anite.com/businesses/handset-testing/our -products.
[8]
3GPP implementation conformance statement. http://www.etsi.org/deliver/etsi_ts/136500_136599/1 3652302/11.03.00_60/ts_13652302v110300p.pdf.
[9]
H. Gruber and M. Holzer. Computational Complexity of NFA Minimization for Finite and Unary Languages. LATA, 8:261--272, 2007.
[10]
Y.-H. E. Yang and V. K. Prasanna. Space-time tradeoff in regular expression matching with semi-deterministic finite automata. In IEEE Infocom 2011.
[11]
M. Sipser. Chapter 1: Regular languages. Introduction to the Theory of Computation, pages 31--90, 1998.
[12]
M. O. Rabin and D. Scott. Finite automata and their decision problems. IBM journal of research and development, 3(2):114--125, 1959.
[13]
M. Sipser. Theorem 1.19 in Introduction to the Theory of Computation, volume 2. Thomson Course Technology Boston, 2006.
[14]
F. R. Moore. On the bounds for state-set size in the proofs of equivalence between deterministic, nondeterministic, and two-way finite automata. IEEE Transactions on computers, 100(10):1211--1214, 1971.
[15]
K. Salomaa and S. Yu. NFA to DFA transformation for finite languages. In International Workshop on Implementing Automata, 1996.
[16]
R. Mandl. Precise bounds associated with the subset construction on various classes of nondeterministic finite automata. In Inf & Sys Sciences, 1973.
[17]
J.-M. Champarnaud, A. Khorsi, and T. Paranthoën. Split and join for minimizing: Brzozowski's algorithm. Stringology, 2002:96--104, 2002.
[18]
Hopcroft. An/n log n algo for minimiz. states in kf in ite automaton. 1971.
[19]
M. Almeida, N. Moreira, and R. Reis. On the performance of automata minimization algorithms. Logic and Theory of Algorithms, page 3, 2007.
[20]
Gómez and et al. DFA minimization: from Brzozowski to Hopcroft. 2013.
[21]
3GPP. TS24.301: Non-Access-Stratum (NAS) protocol for Evolved Packet System (EPS); Stage 3, Jun. 2018.
[22]
3GPP. TS36.331: Radio Resource Control (RRC), 2018.
[23]
Anritsu: verification and test for deployment of LTE. http://www.chinacom.tw/ngn2010/pdf/ngn/Anritsu. pdf.
[24]
Easy Concurrency for C++. https://www.cs.kent.ac.uk/projects/ofa/c++csp/.
[25]
DFA minimization using Hopcroft alogirithm: C++ implementation. http://www.cs.tut.fi/~ava/DFA_minimizer.cc.
[26]
G.-H. Tu, Y. Li, C. Peng, C.-Y. Li, H. Wang, and S. Lu. Control-Plane Protocol Interactions in Cellular Networks. In ACM SIGCOMM, August 2014.
[27]
J. Huang, F. Qian, Y. Guo, Y. Zhou, Q. Xu, Z. M. Mao, S. Sen, and O. Spatscheck. An in-depth study of LTE: effect of network protocol and application behavior on performance. In ACM SIGCOMM, 2013.
[28]
C.-H. Chuang and et al. Performance study for HARQ-- ARQ interaction of LTE. Wireless Communications and Mobile Comp., 10(11):1459--1469, 2010.
[29]
Hussain, Syed Rafiul and Chowdhury, Omar and Mehnaz, Shagufta and Bertino, Elisa. LTEInspector: A Systematic Approach for Adversarial Testing of 4G LTE. In Network and Distributed Systems Security (NDSS) Symposium 2018, 2018.
[30]
A. Shaik and et al. Practical attacks against privacy and availability in 4G/LTE mobile communication systems. 2015.
[31]
R. Subramanian, K. Sandrasegaran, and X. Kong. Benchmarking of real-time LTE network in dynamic environment. In IEEE APCC, 2016.
[32]
H. Zhao and H. Jiang. LTE-M system performance of integrated services based on field test results. In IEEE IMCEC, 2016.
[33]
B. Cui, S. Feng, Q. Xiao, and M. Li. Detection of LTE Protocol Based on Format Fuzz. In IEEE BWCCA, 2015.
[34]
Gerasimenko and et al. Energy and delay analysis of LTE-advanced RACH performance under MTC overload. In IEEE Globecom Workshops, 2012.
[35]
Shi, Jinghao and Lahiri, Shuvendu K and Chandra, Ranveer and Challen, Geoffrey. Wireless protocol validation under uncertainty. In International Conference on Runtime Verification. Springer, 2016.
[36]
Fayaz, Seyed Kaveh and Yu, Tianlong and Tobioka, Yoshiaki and Chaki, Sagar and Sekar, Vyas. BUZZ: Testing Context-Dependent Policies in Stateful Networks. In NSDI, 2016.
[37]
Lee, David and Netravali, Arun N and Sabnani, Krishan K and Sugla, Binay and John, Ajita. Passive testing and applications to network management. In International Conference on Network Protocols. IEEE, 1997.
[38]
Godefroid, and Patrice. Model checking for programming languages using VeriSoft. In Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, 1997.
[39]
Khurshid, Sarfraz and P'sreanu, Corina S and Visser, Willem. Generalized symbolic execution for model checking and testing. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 2003.
[40]
Fiteru-Brotean, Paul and Janssen, Ramon and Vaandrager, Frits. Combining model learning and model checking to analyze TCP implementations. In International Conference on Computer Aided Verification. Springer, 2016.
[41]
Dana Angluin. Learning Regular Sets from Queries and Counterexamples. In IEEE Information and Computation, 1987.
[42]
T.S. Chow. Testing Software Design Modeled by Finite- State Machines. IEEE Transactions on Software Engineering, 1978.
[43]
Petrenko, and Alexandre. Toward testing from finite state machines with symbolic inputs and outputs. Software & Systems Modeling, 2017.
Index Terms
- A Systematic Way to LTE Testing
Recommendations
A generic interoperability testing framework and a systematic development process for automated interoperability testing
Interoperability is a prerequisite to allow users to access systems implemented by different vendors seamlessly. A good baseline to achieve interoperability is the implementation of a common set of standards. However, this is often not sufficient as ...
Lazy Systematic Unit Testing: JWalk versus JUnit
TAICPART-MUTATION '07: Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATIONLazy systematic unit testing with JWalk is compared against regression testing with JUnit, the leading agile testing tool. JWalk produced exhaustive test sets more quickly and recalculated full state and transition coverage, when testing modified or ...
Testing web enabled simulation at scale using metamorphic testing
ICSE-SEIP '21: Proceedings of the 43rd International Conference on Software Engineering: Software Engineering in PracticeWe report on Facebook's deployment of MIA (Metamorphic Interaction Automaton). MIA is used to test Facebook's Web Enabled Simulation, built on a web infrastructure of hundreds of millions of lines of code. MIA tackles the twin problems of test flakiness ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
August 2019
1017 pages
ISBN:9781450361699
DOI:10.1145/3300061
- General Chairs:
- Sharad Agarwal,
- Ben Greenstein,
- Aruna Balasubramanian,
- Program Chairs:
- Shyam Gollakota,
- Xinyu Zhang
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 August 2019
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MobiCom '19
Sponsor:
MobiCom '19: The 25th Annual International Conference on Mobile Computing and Networking
October 21 - 25, 2019
Los Cabos, Mexico
Acceptance Rates
Overall Acceptance Rate 440 of 2,972 submissions, 15%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 590Total Downloads
- Downloads (Last 12 months)31
- Downloads (Last 6 weeks)3
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in