This project contains two different implementations of the LineServer problem:
- LineService uses caching mechanism to facilitate support of the large files and many concurrent users; This version worked just a tiny bit worce for small files with cache size being equal or larger than a size of a file loaded. This version started performing better (better availabily and concurrency) than a basic version starting at ~300Mb. Unline basic version server continue to work while processing 500Mb, 700Mb and 1Gb files. For generating large files I used Fake Apache Log Generator
- LineServiceBasic is the naive implementation which places the entire file into memory upon reading. This implementation was performing better for me on smaller files such as odyssey.txt ~ 600kB. However with large enough files ~1Gb the server just wouldn't start because it was running out of memory
To install requirements in a virtual environment begin by running
$ cd circle
$ virtualenv venv
Activate virtual environment
$ source venv/bin/activate
Followed by installing requirements:
$ pip3 install -r requirements.txt
This is a Flask application which comes with some built-in commands.
To start this application, run
$ export FLASK_APP=src/app.py
$ export FLASK_ENV=development
$ python -m flask run
* Running on http://127.0.0.1:5000/
GET http://127.0.0.1:5000/lines/<line_id>
Test files must be placed inside the src/ folder inside the project and then refferenced as SMALL_FILE or LARGE_FILE in app.py
File odyssey.txt ~ 600kB.
Load testing with Siege
$ siege -c 5 -b -f test_urls.txt -r 5000
Transactions: 25000 hits
Availability: 100.00 %
Elapsed time: 46.63 secs
Data transferred: 1.42 MB
Response time: 0.01 secs
Transaction rate: 536.14 trans/sec
Throughput: 0.03 MB/sec
Concurrency: 4.97
Successful transactions: 25000
Failed transactions: 0
Longest transaction: 19.81
Shortest transaction: 0.00
Transactions: 25000 hits
Availability: 100.00 %
Elapsed time: 44.71 secs
Data transferred: 1.42 MB
Response time: 0.01 secs
Transaction rate: 559.16 trans/sec
Throughput: 0.03 MB/sec
Concurrency: 4.96
Successful transactions: 25000
Failed transactions: 0
Longest transaction: 20.18
Shortest transaction: 0.00
$ siege -c 255 -b -f test_urls.txt -r 1000