Processing /Volumes/workplace/python_env/serve Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting Pillow Using cached Pillow-9.5.0-cp38-cp38-macosx_10_10_x86_64.whl (3.4 MB) Collecting psutil Using cached psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl (243 kB) Collecting packaging Using cached packaging-23.1-py3-none-any.whl (48 kB) Collecting wheel Using cached wheel-0.40.0-py3-none-any.whl (64 kB) Building wheels for collected packages: torchserve Building wheel for torchserve (setup.py): started Building wheel for torchserve (setup.py): finished with status 'done' Created wheel for torchserve: filename=torchserve-0.7.1b20230415-py3-none-any.whl size=23697814 sha256=78c236bc351187ac8db1703071f33fdab23eea5a960722d9ebc6227bdb1a8750 Stored in directory: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/pip-ephem-wheel-cache-51tojcsm/wheels/24/42/60/ce01bb072bdf069a0689515be64114c26b8dc772df749bad0d Successfully built torchserve Installing collected packages: wheel, psutil, Pillow, packaging, torchserve Attempting uninstall: wheel Found existing installation: wheel 0.40.0 Uninstalling wheel-0.40.0: Successfully uninstalled wheel-0.40.0 Attempting uninstall: psutil Found existing installation: psutil 5.9.4 Uninstalling psutil-5.9.4: Successfully uninstalled psutil-5.9.4 Attempting uninstall: Pillow Found existing installation: Pillow 9.5.0 Uninstalling Pillow-9.5.0: Successfully uninstalled Pillow-9.5.0 Attempting uninstall: packaging Found existing installation: packaging 23.1 Uninstalling packaging-23.1: Successfully uninstalled packaging-23.1 Attempting uninstall: torchserve Found existing installation: torchserve 0.7.1b20230415 Uninstalling torchserve-0.7.1b20230415: Successfully uninstalled torchserve-0.7.1b20230415 Successfully installed Pillow-9.5.0 packaging-23.1 psutil-5.9.4 torchserve-0.7.1b20230415 wheel-0.40.0 Processing ./model-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting enum-compat Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB) Building wheels for collected packages: torch-model-archiver Building wheel for torch-model-archiver (setup.py): started Building wheel for torch-model-archiver (setup.py): finished with status 'done' Created wheel for torch-model-archiver: filename=torch_model_archiver-0.7.1b20230415-py3-none-any.whl size=14735 sha256=aa0198f30da3fe9f6f0c632d8557befd6d57723e9468bb3a0492471393b81920 Stored in directory: /Users/lninga/Library/Caches/pip/wheels/40/d0/f5/5019474ce9b424781ec8c156be2dabd8d7526bfba45adc4d1e Successfully built torch-model-archiver Installing collected packages: enum-compat, torch-model-archiver Attempting uninstall: enum-compat Found existing installation: enum-compat 0.0.3 Uninstalling enum-compat-0.0.3: Successfully uninstalled enum-compat-0.0.3 Attempting uninstall: torch-model-archiver Found existing installation: torch-model-archiver 0.7.1b20230415 Uninstalling torch-model-archiver-0.7.1b20230415: Successfully uninstalled torch-model-archiver-0.7.1b20230415 Successfully installed enum-compat-0.0.3 torch-model-archiver-0.7.1b20230415 Processing ./workflow-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: torch-workflow-archiver Building wheel for torch-workflow-archiver (setup.py): started Building wheel for torch-workflow-archiver (setup.py): finished with status 'done' Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.7b20230415-py3-none-any.whl size=12701 sha256=a0214f362d0b29365dc60b899a498dce2df673882c5679a26fbb23eea98af85d Stored in directory: /Users/lninga/Library/Caches/pip/wheels/6a/08/97/e89e9bf08db09591cf7092e9dbfc31fa105e0423bc3c8da6c1 Successfully built torch-workflow-archiver Installing collected packages: torch-workflow-archiver Attempting uninstall: torch-workflow-archiver Found existing installation: torch-workflow-archiver 0.2.7b20230415 Uninstalling torch-workflow-archiver-0.2.7b20230415: Successfully uninstalled torch-workflow-archiver-0.2.7b20230415 Successfully installed torch-workflow-archiver-0.2.7b20230415 WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar ... newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 276ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 35ms 5ms 1ms 291µs 262ms 6ms 373µs 311ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 118ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 613µs (cache) (cache) 115ms 1ms 57µs 118ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 2s time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 2s 1ms 55µs 2s ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 6ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 3ms 1ms 51µs 6ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 25ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 660µs 184µs 242µs 22ms 1ms 67µs 26ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 207µs (cache) (cache) 1ms 1ms 41µs 4ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 657µs 207µs 250µs 14ms 1ms 48µs 18ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 102ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 284µs (cache) (cache) 100ms 1ms 33µs 103ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 6ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 4ms 1ms 31µs 7ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 98ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 96ms 1ms 27µs 99ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1510ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 1507ms 1ms 34µs 1511ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 601µs 178µs 222µs 15ms 1ms 34µs 19ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 7ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 225µs (cache) (cache) 4ms 1ms 40µs 7ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 1ms 1ms 40µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 1ms 1ms 36µs 4ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 7ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 4ms 1ms 36µs 7ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 4ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 1ms 1ms 37µs 5ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 17ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 634µs 176µs 255µs 14ms 1ms 42µs 18ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 124ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 267µs (cache) (cache) 122ms 1ms 28µs 125ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 4ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 1ms 1ms 41µs 4ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 5ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 284µs (cache) (cache) 2ms 1ms 33µs 5ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 216µs (cache) (cache) 1ms 1ms 32µs 4ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 4ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 2ms 1ms 29µs 4ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 15ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 551µs 150µs 221µs 11ms 1ms 30µs 15ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 4ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 295µs (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 5ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 266µs (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 22ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 580µs 161µs 230µs 18ms 1ms 30µs 22ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 32ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 994µs 483µs 133µs 204µs 29ms 1ms 28µs 32ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 105ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 103ms 1ms 25µs 105ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 4ms 1ms 26µs 6ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1443ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 196µs (cache) (cache) 1441ms 1ms 30µs 1443ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 124ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 526µs 136µs 198µs 121ms 1ms 27µs 124ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 4ms 1ms 26µs 6ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 4.2s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 4.1s 1ms 21µs 4.2s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 938µs 201µs (cache) (cache) 3ms 1ms 29µs 5ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 9.6s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 9.5s 3ms 61µs 9.6s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 23ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 473µs (cache) (cache) 20ms 1ms 32µs 24ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 7.1s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 288µs (cache) (cache) 7.1s 1ms 34µs 7.1s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 23ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 20ms 1ms 35µs 24ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 111ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 218µs (cache) (cache) 109ms 1ms 23µs 112ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 161µs (cache) (cache) 3ms 1ms 30µs 6ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 855ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 179µs (cache) (cache) 852ms 1ms 21µs 855ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 163µs (cache) (cache) 3ms 1ms 31µs 6ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 849ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 847ms 983µs 27µs 850ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 939µs 155µs (cache) (cache) 3ms 1ms 26µs 5ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 935ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 154µs (cache) (cache) 933ms 1ms 29µs 936ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 2ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 829µs 133µs (cache) (cache) 1ms 803µs 21µs 2ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 912µs 146µs (cache) (cache) 1ms 973µs 25µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 2ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 950µs 155µs (cache) (cache) 908µs 927µs 25µs 2ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 4ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 189µs (cache) (cache) 1ms 1ms 55µs 4ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 175µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 3ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 165µs (cache) (cache) 1ms 994µs 26µs 3ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 1ms 1ms 28µs 3ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 967µs 154µs (cache) (cache) 1ms 1ms 28µs 3ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 684ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 559µs 182µs 203µs 681ms 1ms 35µs 684ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 18ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 978µs 162µs 230µs 13ms 1ms 36µs 18ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 15ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 521µs 955µs 252µs 11ms 1ms 29µs 15ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 15ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 558µs 167µs 207µs 11ms 1ms 32µs 15ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 207µs (cache) (cache) 1ms 1ms 30µs 4ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 3ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 190µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 172µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 3ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 161µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 13ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 535µs 152µs 249µs 10ms 1ms 26µs 15ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 13ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 990µs 529µs 132µs 199µs 10ms 1ms 25µs 13ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 18ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 16ms 1ms 29µs 18ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 4ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 2ms 1ms 29µs 5ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 13ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 487µs 141µs 207µs 10ms 1ms 31µs 14ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 16ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 994µs 467µs 143µs 185µs 10ms 3ms 30µs 16ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 16ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 464µs 140µs 204µs 13ms 1ms 29µs 16ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 108ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 485µs 143µs 227µs 105ms 1ms 23µs 109ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 5ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 951µs 152µs (cache) (cache) 3ms 1ms 28µs 5ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 105ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 160µs (cache) (cache) 103ms 856µs 22µs 105ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 148µs (cache) (cache) 3ms 1ms 26µs 5ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 125ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 504µs (cache) (cache) 122ms 1ms 41µs 126ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 6ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 4ms 1ms 31µs 6ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 123ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 120ms 995µs 29µs 123ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 6ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 4ms 1ms 37µs 6ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 4ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 276µs (cache) (cache) 1ms 1ms 32µs 4ms ✓ Successful request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 32.8s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 381ms [min: 2ms, max: 9.6s, s.d.: 1398ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 752µs [min: 132µs, max: 1ms, s.d.: 641µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 379ms [min: 908µs, max: 9.5s, s.d.: 1398ms] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman inference Iteration 1/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true 200 OK ★ 52.9s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"my_text_classifier\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 34ms 5ms 1ms 295µs 52.9s 8ms 447µs 53s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier 200 OK ★ 69ms time ★ 353B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.02911965176463127, │ "Sports": 2.943157232948579e-05, │ "Business": 0.9074352383613586, │ "Sci/Tec": 0.06341569125652313 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 9ms 1ms 322µs 469µs 64ms 1ms 41µs 78ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier 200 OK ★ 26ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"my_text_classifier\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 366µs (cache) (cache) 21ms 3ms 106µs 27ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 794B │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier",mo │ del_version="default",} 145.283 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 634µs 176µs 189µs 15ms 1ms 73µs 20ms ✓ Successful GET request Iteration 2/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true 200 OK ★ 58.3s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B │ { │ "status": "Model \"my_text_classifier_scripted\" Ver │ sion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 58.2s 2ms 64µs 58.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier_scripted 200 OK ★ 83ms time ★ 362B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.04559723287820816, │ "Sports": 0.0003771767660509795, │ "Business": 0.08623101562261581, │ "Sci/Tec": 0.8677946329116821 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 307µs (cache) (cache) 80ms 1ms 52µs 85ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier_scripted 200 OK ★ 28ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B │ { │ "status": "Model \"my_text_classifier_scripted\" unr │ egistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 695µs (cache) (cache) 25ms 1ms 49µs 29ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 1.53kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.24kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ _scripted",model_version="default",} 78756.365 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier",mo │ del_version="default",} 145.283 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier_scr │ ipted",model_version="default",} 86.652 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 271µs (cache) (cache) 2ms 1ms 54µs 5ms ✓ Successful GET request Iteration 3/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 2.2s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 249µs (cache) (cache) 2.2s 1ms 45µs 2.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1 200 OK ★ 172ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299158692359924, │ "tiger_cat": 0.24496474862098694, │ "Egyptian_cat": 0.21722468733787537, │ "cougar": 0.002217544475570321 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 368µs (cache) (cache) 169ms 1ms 33µs 174ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 19ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 208µs (cache) (cache) 16ms 1ms 33µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 1.94kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.65kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ _scripted",model_version="default",} 78756.365 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="squeezenet1_1",model_v │ ersion="default",} 68.779 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier",mo │ del_version="default",} 145.283 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier_scr │ ipted",model_version="default",} 86.652 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful GET request Iteration 4/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true 200 OK ★ 5.2s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B │ { │ "status": "Model \"squeezenet1_1_scripted\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 5.2s 1ms 29µs 5.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1_scripted 200 OK ★ 115ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299158692359924, │ "tiger_cat": 0.24496474862098694, │ "Egyptian_cat": 0.21722468733787537, │ "cougar": 0.002217544475570321 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 215µs (cache) (cache) 112ms 2ms 35µs 116ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1_scripted 200 OK ★ 21ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B │ { │ "status": "Model \"squeezenet1_1_scripted\" unregist │ ered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 234µs (cache) (cache) 18ms 1ms 40µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 2.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.08kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1_scri │ pted",model_version="default",} 108914.189 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ _scripted",model_version="default",} 78756.365 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="squeezenet1_1",model_v │ ersion="default",} 68.779 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier",mo │ del_version="default",} 145.283 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="squeezenet1_1_scripted │ ",model_version="default",} 91.314 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="my_text_classifier_scr │ ipted",mode │ (showing 2.05kB/2.08kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 2ms 2ms 52µs 5ms ✓ Successful GET request Iteration 5/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true 200 OK ★ 4.7s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 281µs (cache) (cache) 4.7s 1ms 35µs 4.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161 200 OK ★ 486ms time ★ 111.24kB↑ 418B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 183B │ { │ "tabby": 0.46661895513534546, │ "tiger_cat": 0.46449047327041626, │ "Egyptian_cat": 0.06614057719707489, │ "lynx": 0.0012924439506605268, │ "plastic_bag": 0.00022909804829396307 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 209µs (cache) (cache) 483ms 1ms 30µs 487ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161 200 OK ★ 19ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 252µs (cache) (cache) 16ms 1ms 44µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 2.77kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.48kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1_scri │ pted",model_version="default",} 108914.189 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ _scripted",model_version="default",} 78756.365 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="squeezenet1_1",model_v │ ersion="default",} 68.779 │ ts_queue_latency_microseconds{uuid="cfcf5154-bf33-4d64 │ -9467-8bf0d0e1b059",model_name="densenet161",model_ver │ sion="default",} 92.619 │ ts_ │ (showing 2.05kB/2.48kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 264µs (cache) (cache) 2ms 1ms 36µs 5ms ✓ Successful GET request Iteration 6/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true 200 OK ★ 7.7s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B │ { │ "status": "Model \"alexnet\" Version: 1.0 registered │ with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 7.7s 1ms 38µs 7.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet 200 OK ★ 118ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847333908081055, │ "tiger_cat": 0.25793954730033875, │ "Egyptian_cat": 0.24254852533340454, │ "lynx": 0.16879378259181976, │ "tiger": 0.006487946026027203 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 203µs (cache) (cache) 115ms 2ms 40µs 120ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet 200 OK ★ 39ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B │ { │ "status": "Model \"alexnet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 348µs (cache) (cache) 37ms 1ms 32µs 40ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 3.16kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.86kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1_scri │ pted",model_version="default",} 108914.189 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="alexnet",model_ver │ sion="default",} 111437.699 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ _scripted",model_version="default",} 78756.365 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_l │ (showing 2.05kB/2.86kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 239µs (cache) (cache) 2ms 2ms 58µs 6ms ✓ Successful GET request Iteration 7/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true 200 OK ★ 1m 32.3s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B │ { │ "status": "Model \"alexnet_scripted\" Version: 1.0 r │ egistered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 1m 32.3s 1ms 33µs 1m 32.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet_scripted 200 OK ★ 162ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847333908081055, │ "tiger_cat": 0.25793954730033875, │ "Egyptian_cat": 0.24254852533340454, │ "lynx": 0.16879378259181976, │ "tiger": 0.006487946026027203 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 234µs (cache) (cache) 158ms 1ms 38µs 163ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet_scripted 200 OK ★ 21ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B │ { │ "status": "Model \"alexnet_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 259µs (cache) (cache) 18ms 1ms 35µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 3.57kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.28kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1_scri │ pted",model_version="default",} 108914.189 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="alexnet_scripted", │ model_version="default",} 155483.915 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="alexnet",model_ver │ sion="default",} 111437.699 │ ts_inference │ (showing 2.05kB/3.28kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 266µs (cache) (cache) 2ms 1ms 43µs 6ms ✓ Successful GET request Iteration 8/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 3.2s 1ms 24µs 3.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18 200 OK ★ 104ms time ★ 111.24kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.40966302156448364, │ "tiger_cat": 0.3467046916484833, │ "Egyptian_cat": 0.13002900779247284, │ "lynx": 0.023919543251395226, │ "bucket": 0.011532166972756386 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 152µs (cache) (cache) 102ms 1ms 29µs 105ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18 200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 598µs (cache) (cache) 17ms 1ms 36µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 3.96kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.67kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1_scri │ pted",model_version="default",} 108914.189 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="resnet-18",model_v │ ersion="default",} 99564.202 │ ts_inference_latency_microsecon │ (showing 2.05kB/3.67kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 1ms 1ms 35µs 5ms ✓ Successful GET request Iteration 9/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true 200 OK ★ 21s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"resnet-18_scripted\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 222µs (cache) (cache) 21s 1ms 27µs 21s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18_scripted 200 OK ★ 205ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.40966302156448364, │ "tiger_cat": 0.3467046916484833, │ "Egyptian_cat": 0.13002900779247284, │ "lynx": 0.023919543251395226, │ "bucket": 0.011532166972756386 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 199µs (cache) (cache) 202ms 1ms 31µs 206ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18_scripted 200 OK ★ 19ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"resnet-18_scripted\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 16ms 1ms 44µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 4.38kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.09kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1_scri │ pted",model_version="default",} 108914.189 │ ts_inference_latency_microseconds{ │ (showing 2.05kB/4.09kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 249µs (cache) (cache) 1ms 1ms 42µs 6ms ✓ Successful GET request Iteration 10/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true 200 OK ★ 15.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"vgg16\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 222µs (cache) (cache) 15.2s 1ms 38µs 15.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16 200 OK ★ 488ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.44697263836860657, │ "tabby": 0.4408797323703766, │ "Egyptian_cat": 0.05904562026262283, │ "tiger": 0.020596399903297424, │ "lynx": 0.009934580884873867 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 305µs (cache) (cache) 485ms 1ms 34µs 490ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16 200 OK ★ 26ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"vgg16\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 260µs (cache) (cache) 23ms 1ms 40µs 27ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 10ms time ★ 233B↑ 4.76kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.47kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="my_text_classifier │ ",model_version="default",} 40196.695 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467- │ (showing 2.05kB/4.47kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 1ms (cache) (cache) 2ms 5ms 53µs 11ms ✓ Successful GET request Iteration 11/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true 200 OK ★ 2m 53.9s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"vgg16_scripted\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 284µs (cache) (cache) 2m 53.9s 1ms 32µs 2m 53.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16_scripted 200 OK ★ 581ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.44697263836860657, │ "tabby": 0.4408797323703766, │ "Egyptian_cat": 0.05904562026262283, │ "tiger": 0.020596399903297424, │ "lynx": 0.009934580884873867 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 228µs (cache) (cache) 578ms 1ms 34µs 583ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16_scripted 200 OK ★ 65ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"vgg16_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 62ms 1ms 39µs 65ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 5.17kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.88kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet161",model │ _version="default",} 479969.073 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",mo │ (showing 2.05kB/4.88kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 248µs (cache) (cache) 2ms 2ms 87µs 7ms ✓ Successful GET request Iteration 12/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 2.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 278µs (cache) (cache) 4s 1ms 31µs 4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist 200 OK ★ 22ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 225µs (cache) (cache) 19ms 1ms 35µs 23ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 22ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 188µs (cache) (cache) 19ms 1ms 31µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 5.55kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.26kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_version="default",} 165056.424 │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="densenet1 │ (showing 2.05kB/5.26kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 1ms 1ms 54µs 4ms ✓ Successful GET request Iteration 13/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"mnist_scripted\" Version: 2.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 279µs (cache) (cache) 3.3s 1ms 27µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist_scripted 200 OK ★ 66ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 194µs (cache) (cache) 64ms 1ms 27µs 67ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist_scripted 200 OK ★ 21ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"mnist_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 162µs (cache) (cache) 18ms 1ms 38µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 5.96kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.67kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="cfcf5154-bf33- │ 4d64-9467-8bf0d0e1b059",model_name="squeezenet1_1",mod │ el_ │ (showing 2.05kB/5.67kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 255µs (cache) (cache) 1ms 1ms 191µs 8ms ✓ Successful GET request Iteration 14/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true 200 OK ★ 6s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"fastrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 276µs (cache) (cache) 6s 1ms 37µs 6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fastrcnn 200 OK ★ 5.2s time ★ 289.21kB↑ 3kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.76kB │ [ │ { │ "person": [ │ 167.39585876464844, │ 57.20302200317383, │ 301.3599853515625, │ 436.7974548339844 │ ], │ "score": 0.9995203018188477 │ }, │ { │ "person": [ │ 89.54701232910156, │ 64.8357162475586, │ 191.42431640625, │ 446.76068115234375 │ ], │ "score": 0.9994968175888062 │ }, │ { │ "person": [ │ 362.37823486328125, │ 161.84130859375, │ 515.5023193359375, │ 385.2898254394531 │ ], │ "score": 0.9977061748504639 │ }, │ { │ "handbag": [ │ 67.3762435913086, │ 277.5755920410156, │ 111.67390441894531, │ 400.2067565917969 │ ], │ "score": 0.9924989342689514 │ }, │ { │ "handbag": [ │ 228.6824951171875, │ 146.00692749023438, │ 303.55120849609375, │ 231.08848571777344 │ ], │ "score": 0.9922404289245605 │ }, │ { │ "handbag": [ │ 379.411376953125, │ 259.9578552246094, │ 419.0797424316406, │ 317.9610290527344 │ ], │ "score": 0.9898616075515747 │ }, │ { │ "person": [ │ 518.49609375, │ 149.73033142089844, │ 636.6343994140625, │ 365.4130554199219 │ ], │ "score": 0.9821102023124695 │ }, │ { │ "bench": [ │ 269.0818176269531, │ 217.34210205078125, │ 423.77008056640625, │ 390.37847900390625 │ ], │ "score": 0.9573417901992798 │ }, │ { │ "person": [ │ 539.6298217773438, │ 157.7587127685547, │ 616.1531372070312, │ 253.1112823486328 │ ], │ "score": 0.8995429873466492 │ }, │ { │ "person": [ │ 477.0623779296875, │ 147.8089141845703, │ 610.77734375, │ 296.8367004394531 │ ], │ "score": 0.8751581311225891 │ }, │ { │ "bench": [ │ 286.0877990722656, │ 216.53292846679688, │ 550.7698364257812, │ 383.1821594238281 │ ], │ "score": 0.8436439037322998 │ }, │ { │ "person": [ │ 627.4741821289062, │ 177.05836486816406, │ 640.0, │ 247.72146606445312 │ ], │ "score": 0.8257301449775696 │ }, │ { │ "bench": [ │ 88.76304626464844, │ 226.1649932861328, │ 563.6138916015625, │ 421. │ (showing 2.05kB/2.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 232µs (cache) (cache) 5.2s 1ms 34µs 5.2s ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fastrcnn 200 OK ★ 20ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"fastrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 181µs (cache) (cache) 17ms 1ms 34µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 6.35kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.06kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fastrcnn",model_version= │ "default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microsec │ (showing 2.05kB/6.06kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 1ms 1ms 44µs 4ms ✓ Successful GET request Iteration 15/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true 200 OK ★ 6s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"maskrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 189µs (cache) (cache) 6s 1ms 34µs 6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/maskrcnn 200 OK ★ 6s time ★ 289.21kB↑ 3kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.76kB │ [ │ { │ "person": [ │ 169.50631713867188, │ 49.98546600341797, │ 300.8945617675781, │ 442.4545593261719 │ ], │ "score": 0.9995434880256653 │ }, │ { │ "person": [ │ 90.4118423461914, │ 66.7905044555664, │ 194.1930389404297, │ 437.27777099609375 │ ], │ "score": 0.9993956089019775 │ }, │ { │ "person": [ │ 362.3172912597656, │ 158.1734619140625, │ 521.2589721679688, │ 385.7226867675781 │ ], │ "score": 0.9952279329299927 │ }, │ { │ "handbag": [ │ 68.57540893554688, │ 279.3182067871094, │ 111.15328979492188, │ 400.9166564941406 │ ], │ "score": 0.9938817024230957 │ }, │ { │ "person": [ │ 474.01568603515625, │ 147.34800720214844, │ 638.1206665039062, │ 364.6508483886719 │ ], │ "score": 0.9897462129592896 │ }, │ { │ "handbag": [ │ 225.59584045410156, │ 142.90005493164062, │ 302.48638916015625, │ 230.3284149169922 │ ], │ "score": 0.9891214966773987 │ }, │ { │ "handbag": [ │ 380.2604675292969, │ 259.20135498046875, │ 419.53656005859375, │ 318.27728271484375 │ ], │ "score": 0.9688040614128113 │ }, │ { │ "bench": [ │ 273.48565673828125, │ 217.4885711669922, │ 441.06439208984375, │ 396.241455078125 │ ], │ "score": 0.961753249168396 │ }, │ { │ "person": [ │ 541.289794921875, │ 156.6612548828125, │ 619.938232421875, │ 249.45277404785156 │ ], │ "score": 0.8177030086517334 │ }, │ { │ "person": [ │ 362.9620666503906, │ 163.8992462158203, │ 500.7699279785156, │ 293.9123840332031 │ ], │ "score": 0.8016932010650635 │ }, │ { │ "chair": [ │ 455.20855712890625, │ 207.54006958007812, │ 491.0853271484375, │ 274.64752197265625 │ ], │ "score": 0.77582848072052 │ }, │ { │ "person": [ │ 549.1541137695312, │ 177.42166137695312, │ 640.0, │ 364.5395202636719 │ ], │ "score": 0.7176401615142822 │ }, │ { │ "person": [ │ 626.230712890625, │ 178.65342712402344, │ 640.0, │ 24 │ (showing 2.05kB/2.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 6s 1ms 33µs 6s ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/maskrcnn 200 OK ★ 19ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"maskrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 16ms 1ms 35µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 6.74kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.45kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fastrcnn",model_version= │ "default",} 1.0 │ (showing 2.05kB/6.45kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 353µs (cache) (cache) 2ms 1ms 52µs 6ms ✓ Successful GET request Iteration 16/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true 200 OK ★ 7.3s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"fcn_resnet_101\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 199µs (cache) (cache) 7.3s 1ms 32µs 7.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101 200 OK ★ 1409ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 237µs (cache) (cache) 1402ms 5ms 1ms 1412ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101 200 OK ★ 22ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"fcn_resnet_101\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 300µs (cache) (cache) 19ms 1ms 75µs 24ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 7.15kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.86kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default" │ (showing 2.05kB/6.86kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 1ms 1ms 53µs 4ms ✓ Successful GET request Iteration 17/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 1m 22s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B │ { │ "status": "Model \"fcn_resnet_101_scripted\" Version │ : 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1m 22s 3ms 51µs 1m 22s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101_scripted 200 OK ★ 1759ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 516µs (cache) (cache) 1649ms 108ms 1ms 1765ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101_scripted 200 OK ★ 46ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B │ { │ "status": "Model \"fcn_resnet_101_scripted\" unregis │ tered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 274µs (cache) (cache) 43ms 1ms 62µs 48ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 7.59kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.29kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16",model_v │ (showing 2.05kB/7.29kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 1ms 1ms 46µs 5ms ✓ Successful GET request Iteration 18/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true 200 OK ★ 8.2s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" Vers │ ion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 8.2s 3ms 74µs 8.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager 200 OK ★ 1519ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954884648323059 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949429631233215 │ ], │ [ │ 0.0, │ 0.9946452379226685 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945646524429321 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948964715003967 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949837923049927 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.995111525 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 282µs (cache) (cache) 1513ms 4ms 1ms 1523ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager 200 OK ★ 23ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" unre │ gistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 272µs (cache) (cache) 20ms 1ms 62µs 25ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.03kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.74kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_eag │ er",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet │ (showing 2.05kB/7.74kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 276µs (cache) (cache) 1ms 1ms 41µs 5ms ✓ Successful GET request Iteration 19/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 3m 11.8s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 3m 11.7s 2ms 54µs 3m 11.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted 200 OK ★ 1687ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954884648323059 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949429631233215 │ ], │ [ │ 0.0, │ 0.9946452379226685 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945646524429321 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948964715003967 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949837923049927 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.995111525 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 502µs (cache) (cache) 1682ms 4ms 315µs 1693ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted 200 OK ★ 24ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 252µs (cache) (cache) 21ms 1ms 63µs 26ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.48kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.19kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="dee │ (showing 2.05kB/8.19kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 292µs (cache) (cache) 1ms 1ms 41µs 5ms ✓ Successful GET request Iteration 20/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true 200 OK ★ 7.8s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 233µs (cache) (cache) 7.8s 1ms 33µs 7.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet152 200 OK ★ 431ms time ★ 111.24kB↑ 411B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 176B │ { │ "tiger_cat": 0.5798617601394653, │ "tabby": 0.38344141840934753, │ "Egyptian_cat": 0.03421143442392349, │ "lynx": 0.0005819805082865059, │ "quilt": 0.00027331963065080345 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 212µs (cache) (cache) 428ms 1ms 33µs 432ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet152 200 OK ★ 23ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 20ms 1ms 33µs 23ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.88kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.59kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="res │ (showing 2.05kB/8.59kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 273µs (cache) (cache) 1ms 1ms 42µs 5ms ✓ Successful GET request Iteration 21/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true 200 OK ★ 1m 58.8s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "status": "Model \"resnet-152-batch_scripted\" Versi │ on: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 191µs (cache) (cache) 1m 58.8s 2ms 50µs 1m 58.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-152-batch_scripted 200 OK ★ 807ms time ★ 111.26kB↑ 411B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 176B │ { │ "tiger_cat": 0.5798617601394653, │ "tabby": 0.38344141840934753, │ "Egyptian_cat": 0.03421143442392349, │ "lynx": 0.0005819805082865059, │ "quilt": 0.00027331963065080345 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 270µs (cache) (cache) 805ms 1ms 31µs 809ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-152-batch_scripted 200 OK ★ 20ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B │ { │ "status": "Model \"resnet-152-batch_scripted\" unreg │ istered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 227µs (cache) (cache) 17ms 2ms 34µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 9.32kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.03kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-152-batch_scripte │ d",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name=" │ (showing 2.05kB/9.03kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 250µs (cache) (cache) 1ms 1ms 40µs 5ms ✓ Successful GET request Iteration 22/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true 200 OK ★ 1m 50.8s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "status": "Model \"distill_bert_qa_eager\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 1m 50.8s 1ms 36µs 1m 50.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/distill_bert_qa_eager 200 OK ★ 248ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 79B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B │ a nice puppet └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 247µs (cache) (cache) 245ms 1ms 31µs 249ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/distill_bert_qa_eager 200 OK ★ 19ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Model \"distill_bert_qa_eager\" unregiste │ red" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 206µs (cache) (cache) 16ms 1ms 36µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 9.75kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.46kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-152-batch_scripte │ d",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name=" │ (showing 2.05kB/9.46kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 313µs (cache) (cache) 2ms 1ms 55µs 7ms ✓ Successful GET request Iteration 23/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true 200 OK ★ 2m 38.3s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" Version: 1.0 registered with 1 initial worke │ rs" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 2m 38.3s 1ms 79µs 2m 38.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_token_classification_no_torchscript 200 OK ★ 382ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB │ [ │ [ │ "[CLS]", │ "B-LOC" │ ], │ [ │ "bloomberg", │ "O" │ ], │ [ │ "has", │ "I-ORG" │ ], │ [ │ "decided", │ "B-PER" │ ], │ [ │ "to", │ "O" │ ], │ [ │ "publish", │ "I-PER" │ ], │ [ │ "a", │ "I-MISC" │ ], │ [ │ "new", │ "B-PER" │ ], │ [ │ "report", │ "O" │ ], │ [ │ "on", │ "O" │ ], │ [ │ "global", │ "B-PER" │ ], │ [ │ "economic", │ "O" │ ], │ [ │ "situation", │ "B-LOC" │ ], │ [ │ ".", │ "B-PER" │ ], │ [ │ "[SEP]", │ "B-PER" │ ], │ [ │ "[PAD]", │ "I-ORG" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ (showing 2.05kB/5.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 254µs (cache) (cache) 380ms 1ms 34µs 384ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_token_classification_no_torchscript 200 OK ★ 26ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 339µs (cache) (cache) 23ms 1ms 38µs 26ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 15ms time ★ 233B↑ 10.23kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.94kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-152-batch_scripte │ d",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name=" │ (showing 2.05kB/9.94kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 573µs (cache) (cache) 12ms 1ms 43µs 19ms ✓ Successful GET request Iteration 24/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true 200 OK ★ 2m 35.2s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"bert_seqc_without_torchscript\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 212µs (cache) (cache) 2m 35.2s 1ms 35µs 2m 35.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_seqc_without_torchscript 200 OK ★ 430ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B │ Not Accepted └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 215µs (cache) (cache) 427ms 1ms 32µs 432ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_seqc_without_torchscript 200 OK ★ 24ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"bert_seqc_without_torchscript\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 21ms 1ms 53µs 24ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 10.69kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.39kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="deeplabv3_resnet_101_eag │ er",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="cfcf5154-bf33-4d64-9 │ 467-8bf0d0e1b059",model_name="bert_seqc_without_torchs │ cript │ (showing 2.05kB/10.39kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 3ms 1ms 50µs 6ms ✓ Successful GET request ┌─────────────────────────┬──────────────────────┬──────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ iterations │ 24 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ requests │ 96 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ test-scripts │ 96 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ assertions │ 113 │ 0 │ ├─────────────────────────┴──────────────────────┴──────────────────────┤ │ total run duration: 22m 4s │ ├───────────────────────────────────────────────────────────────────────┤ │ total data received: 10.04MB (approx) │ ├───────────────────────────────────────────────────────────────────────┤ │ average response time: 13.7s [min: 4ms, max: 3m 11.8s, s.d.: 39.1s] │ ├───────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 712µs [min: 176µs, max: 1ms, s.d.: 581µs] │ ├───────────────────────────────────────────────────────────────────────┤ │ average first byte time: 13.7s [min: 1ms, max: 3m 11.7s, s.d.: 39.1s] │ └───────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman inference → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true 200 OK ★ 41.5s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B │ { │ "status": "Model \"densenet161_scripted\" Version: 1 │ .0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 5ms 1ms 291µs 41.5s 6ms 368µs 41.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161_scripted 200 OK ★ 1061ms time ★ 111.25kB↑ 418B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 183B │ { │ "tabby": 0.46661895513534546, │ "tiger_cat": 0.46449047327041626, │ "Egyptian_cat": 0.06614057719707489, │ "lynx": 0.0012924439506605268, │ "plastic_bag": 0.00022909804829396307 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms 325µs 257µs 1056ms 2ms 55µs 1067ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161_scripted 200 OK ★ 21ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B │ { │ "status": "Model \"densenet161_scripted\" unregister │ ed" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 427µs (cache) (cache) 17ms 2ms 107µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 20ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 802B │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="fd79bb6c-bf88- │ 49be-8dc4-ebca36cf4a05",model_name="densenet161_script │ ed",model_version="default",} 1030655.487 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="fd79bb6c-bf88-49be │ -8dc4-ebca36cf4a05",model_name="densenet161_scripted", │ model_version="default",} 152.437 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="fd79bb6c-bf88-49be-8 │ dc4-ebca36cf4a05",model_name="densenet161_scripted",mo │ del_version="default",} 1.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 1ms 211µs 231µs 15ms 1ms 76µs 20ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 42.8s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 1.14kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 10.6s [min: 20ms, max: 41.5s, s.d.: 17.8s] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 712µs [min: 211µs, max: 1ms, s.d.: 559µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 10.6s [min: 15ms, max: 41.5s, s.d.: 17.8s] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 174ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 29ms 10ms 1ms 3ms 119ms 33ms 5ms 355µs 203ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 51ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 180µs 380µs 30ms 16ms 1ms 66µs 51ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 30ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 752µs 127µs 317µs 23ms 3ms 1ms 55µs 30ms ✓ Status code is 200 → HTTPS Register Model - SqueezeNet POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 456µs (cache) (cache) (cache) 2s 1ms 82µs 2s ✓ Successful POST request → HTTPS Get SqueezeNet Model Description GET https://localhost:8444/models/squeezenet1_1 200 OK ★ 6ms time ★ 246B↑ 753B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 486B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-04-15T20:33:24.186Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 31068, │ "gpu": false, │ "gpuUsage": "N/A" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 274µs (cache) (cache) (cache) 4ms 1ms 58µs 7ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 5ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 285µs (cache) (cache) (cache) 2ms 1ms 54µs 5ms ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 5ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 280µs (cache) (cache) (cache) 2ms 1ms 40µs 5ms ✓ Successful PUT request → HTTPS - Inference - SqueezeNet POST https://localhost:8443/predictions/squeezenet1_1 200 OK ★ 66ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299158692359924, │ "tiger_cat": 0.24496474862098694, │ "Egyptian_cat": 0.21722468733787537, │ "cougar": 0.002217544475570321 │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 743µs (cache) (cache) (cache) 63ms 1ms 41µs 71ms ✓ Status code is 200 → HTTPS UnRegister Model SqueezeNet DELETE https://localhost:8444/models/squeezenet1_1 200 OK ★ 21ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 256µs (cache) (cache) (cache) 18ms 1ms 33µs 21ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 9 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 2.7s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 85.45kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 271ms [min: 5ms, max: 2s, s.d.: 644ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 532µs [min: 127µs, max: 1ms, s.d.: 536µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 247ms [min: 2ms, max: 2s, s.d.: 649ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 257ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 5ms 1ms 323µs 243ms 5ms 335µs 289ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 135ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 675µs (cache) (cache) 132ms 1ms 70µs 136ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 2.1s time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 287µs (cache) (cache) 2.1s 1ms 51µs 2.1s ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 5ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 2ms 1ms 53µs 5ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 19ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 670µs 195µs 262µs 15ms 1ms 66µs 19ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 216µs (cache) (cache) 1ms 1ms 42µs 4ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 585µs 170µs 223µs 14ms 1ms 47µs 17ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 97ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 303µs (cache) (cache) 95ms 1ms 34µs 98ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 6ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 216µs (cache) (cache) 4ms 1ms 30µs 6ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 101ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 196µs (cache) (cache) 98ms 1ms 28µs 101ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1381ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 192µs (cache) (cache) 1379ms 1ms 33µs 1381ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 556µs 166µs 208µs 13ms 1ms 34µs 17ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 6ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 3ms 1ms 38µs 6ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 1ms 1ms 39µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 1ms 1ms 32µs 4ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 7ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 5ms 1ms 36µs 7ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 4ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 15ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 643µs 173µs 256µs 12ms 1ms 31µs 15ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 101ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 238µs (cache) (cache) 99ms 967µs 23µs 101ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 4ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 173µs (cache) (cache) 2ms 1ms 39µs 4ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 176µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 209µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 195µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 14ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 488µs 144µs 198µs 11ms 1ms 29µs 14ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 4ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 199µs (cache) (cache) 1ms 1ms 30µs 4ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 977µs 198µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 20ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 540µs 154µs 246µs 17ms 1ms 28µs 20ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 28ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 549µs 145µs 202µs 25ms 1ms 28µs 28ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 106ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 104ms 931µs 23µs 107ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 913µs 176µs (cache) (cache) 3ms 1ms 26µs 5ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1263ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 1261ms 1ms 31µs 1263ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 113ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 525µs 138µs 204µs 110ms 973µs 24µs 113ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 4ms 1ms 28µs 6ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 4.3s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 199µs (cache) (cache) 4.3s 2ms 52µs 4.3s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 8ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 365µs (cache) (cache) 5ms 1ms 30µs 8ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 7.9s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 7.9s 1ms 31µs 7.9s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 23ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 20ms 1ms 37µs 24ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 7.8s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 232µs (cache) (cache) 7.8s 1ms 33µs 7.8s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 17ms 1ms 34µs 20ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 134ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 130ms 1ms 42µs 134ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 429µs (cache) (cache) 3ms 1ms 30µs 7ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 841ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 209µs (cache) (cache) 839ms 851µs 21µs 842ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 6ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 3ms 1ms 32µs 6ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 845ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 843ms 869µs 24µs 845ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 826µs 164µs (cache) (cache) 2ms 1ms 28µs 5ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 825ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 823ms 896µs 27µs 826ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 3ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 177µs (cache) (cache) 1ms 841µs 23µs 3ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 169µs (cache) (cache) 1ms 891µs 28µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 275µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 4ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 239µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 3ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 989µs 161µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 1ms 1ms 33µs 3ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 174µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 194µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 1793ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 565µs 681µs 219µs 1789ms 975µs 28µs 1793ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 11ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 374µs 108µs 168µs 8ms 931µs 27µs 11ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 13ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 964µs 451µs 141µs 205µs 10ms 965µs 26µs 13ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 13ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 962µs 430µs 130µs 176µs 10ms 1ms 33µs 13ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 987µs 159µs (cache) (cache) 1ms 1ms 28µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 3ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 991µs 172µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 3ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 971µs 163µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 3ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 938µs 152µs (cache) (cache) 1ms 934µs 25µs 3ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 13ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 442µs 125µs 206µs 10ms 1ms 26µs 14ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 13ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 459µs 136µs 189µs 10ms 1ms 27µs 14ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 15ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 908µs 147µs (cache) (cache) 12ms 1ms 32µs 15ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 5ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 888µs (cache) (cache) 2ms 1ms 32µs 5ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 13ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 494µs 148µs 226µs 10ms 1ms 29µs 13ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 16ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 589µs 164µs 244µs 11ms 3ms 30µs 16ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 8ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 999µs 471µs 145µs 234µs 5ms 1ms 31µs 8ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 994µs 166µs (cache) (cache) 1ms 992µs 29µs 3ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 107ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 490µs 768µs 230µs 104ms 929µs 24µs 107ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 5ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 3ms 1ms 29µs 5ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 101ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 99ms 870µs 23µs 101ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 905µs 142µs (cache) (cache) 3ms 987µs 26µs 5ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 101ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 165µs (cache) (cache) 99ms 879µs 23µs 101ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 3ms 941µs 27µs 5ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 101ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 164µs (cache) (cache) 99ms 937µs 29µs 101ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 957µs 162µs (cache) (cache) 3ms 986µs 26µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 956µs 159µs (cache) (cache) 1ms 960µs 26µs 3ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 32.7s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 380ms [min: 3ms, max: 7.9s, s.d.: 1331ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 751µs [min: 108µs, max: 1ms, s.d.: 640µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 378ms [min: 1ms, max: 7.9s, s.d.: 1331ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kf_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 4ms 1ms 243µs 2s 6ms 380µs 2.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v1/models/mnist:predict 200 OK ★ 40ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms 280µs 445µs 35ms 1ms 60µs 45ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v1/models/mnist:explain 200 OK ★ 82ms time ★ 680B↑ 25.39kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.15kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.004570948738580575, │ 0.006216969354192105, │ 0.008197565590552321, │ 0.009563574614432992, │ 0.008999274562253209, │ 0.009673474929186786, │ 0.0075999051701512495, │ 0.006361380888646387, │ 0.005768828652797567, │ 0.004394465856527532, │ 0.004948218006895951, │ 0.005273460550333246, │ 0.005523799719808455, │ 0.007789356414668216, │ 0.008759363342655755, │ 0.00430480516324295, │ 0.010970579130997414, │ 0.0032484391606445308, │ 0.0059980330139177435, │ 0.0037543659205210495, │ 0.002765290850626197, │ 0.004314086250589837, │ 0.0014008573838089065, │ 0.004841846519426633, │ 0.00063748405852181, │ 0.0018558539683395966, │ -0.0008280458164171088, │ -0.0 │ ], │ [ │ 0.0016625159121928454, │ 0.00444379180663651, │ 0.01238784029399793, │ 0.009450843952942047, │ 0.016143820706701874, │ 0.007797502042012509, │ 0.013942240780593617, │ 0.007557430476889445, │ 0.005479090571418102, │ 0.009751320337132724, │ 0.004764407385073551, │ 0.007292147658128949, │ 0.01179799937538059, │ 0.006462684558412935, │ 0.003383214208849701, │ 0.009225058866604059, │ 0.001675070306666094, │ 0.0073623047131398675, │ 0.005270058076607665, │ 0.005453598185890411, │ 0.004342725772494382, │ 0.005856133228586909, │ 0.012168384676170195, │ 0.009902719402082216, │ 0.009813113253425626, │ 0.003442758203666522, │ 0.0022924284776334007, │ -0.0 │ ], │ [ │ 0.011528255254417058, │ 0.00914698931587443, │ 0.02226541986175408, │ 0.017558217234640273, │ 0.02477055450087839, │ 0.021412693644484452, │ │ (showing 2.05kB/25.15kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 502µs (cache) (cache) 78ms 1ms 85µs 83ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 22ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 286µs (cache) (cache) 19ms 1ms 50µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 763B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="efa49d29-5957-4ebc │ -91c5-273a2578307e",model_name="mnist",model_version=" │ default",} 219.377 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="efa49d29-5957-4ebc-9 │ 1c5-273a2578307e",model_name="mnist",model_version="de │ fault",} 2.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="efa49d29-5957- │ 4ebc-91c5-273a2578307e",model_name="mnist",model_versi │ on="default",} 91035.04299999999 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 673µs 182µs 257µs 16ms 1ms 64µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 2.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 26.07kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 450ms [min: 19ms, max: 2s, s.d.: 820ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 643µs [min: 182µs, max: 1ms, s.d.: 485µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 445ms [min: 16ms, max: 2s, s.d.: 816ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kf_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 179ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 33ms 7ms 1ms 580µs 124ms 38ms 7ms 431µs 212ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 52ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 195µs 370µs 33ms 14ms 1ms 68µs 52ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 39ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 191µs 443µs 31ms 3ms 1ms 80µs 39ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 492µs (cache) (cache) (cache) 2s 1ms 76µs 2s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 6ms time ★ 238B↑ 737B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 470B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-04-15T20:35:05.921Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 31214, │ "gpu": false, │ "gpuUsage": "N/A" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 290µs (cache) (cache) (cache) 3ms 1ms 54µs 6ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 2.8s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 275µs (cache) (cache) (cache) 2.8s 1ms 51µs 2.8s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 4ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 291µs (cache) (cache) (cache) 2ms 1ms 38µs 5ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v1/models/mnist:predict 200 OK ★ 24ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 639µs (cache) (cache) (cache) 21ms 1ms 40µs 29ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v1/models/mnist:explain 200 OK ★ 56ms time ★ 680B↑ 25.39kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.15kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.004570948738580575, │ 0.006216969354192105, │ 0.008197565590552321, │ 0.009563574614432992, │ 0.008999274562253209, │ 0.009673474929186786, │ 0.0075999051701512495, │ 0.006361380888646387, │ 0.005768828652797567, │ 0.004394465856527532, │ 0.004948218006895951, │ 0.005273460550333246, │ 0.005523799719808455, │ 0.007789356414668216, │ 0.008759363342655755, │ 0.00430480516324295, │ 0.010970579130997414, │ 0.0032484391606445308, │ 0.0059980330139177435, │ 0.0037543659205210495, │ 0.002765290850626197, │ 0.004314086250589837, │ 0.0014008573838089065, │ 0.004841846519426633, │ 0.00063748405852181, │ 0.0018558539683395966, │ -0.0008280458164171088, │ -0.0 │ ], │ [ │ 0.0016625159121928454, │ 0.00444379180663651, │ 0.01238784029399793, │ 0.009450843952942047, │ 0.016143820706701874, │ 0.007797502042012509, │ 0.013942240780593617, │ 0.007557430476889445, │ 0.005479090571418102, │ 0.009751320337132724, │ 0.004764407385073551, │ 0.007292147658128949, │ 0.01179799937538059, │ 0.006462684558412935, │ 0.003383214208849701, │ 0.009225058866604059, │ 0.001675070306666094, │ 0.0073623047131398675, │ 0.005270058076607665, │ 0.005453598185890411, │ 0.004342725772494382, │ 0.005856133228586909, │ 0.012168384676170195, │ 0.009902719402082216, │ 0.009813113253425626, │ 0.003442758203666522, │ 0.0022924284776334007, │ -0.0 │ ], │ [ │ 0.011528255254417058, │ 0.00914698931587443, │ 0.02226541986175408, │ 0.017558217234640273, │ 0.02477055450087839, │ 0.021412693644484452, │ │ (showing 2.05kB/25.15kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 272µs (cache) (cache) (cache) 53ms 1ms 37µs 57ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 91ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 268µs (cache) (cache) (cache) 88ms 1ms 36µs 91ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5.7s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 110.41kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 540ms [min: 4ms, max: 2.8s, s.d.: 984ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 544µs [min: 191µs, max: 1ms, s.d.: 496µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 517ms [min: 2ms, max: 2.8s, s.d.: 994ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kfv2_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 30ms 4ms 1ms 492µs 2s 6ms 386µs 2.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v2/models/mnist/infer 200 OK ★ 32ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms 305µs 452µs 28ms 1ms 69µs 38ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v2/models/mnist/explain 200 OK ★ 60ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752831520071, │ -0.0002261287725670648, │ -0.0001273413024972421, │ 0.005648369417700337, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 544µs (cache) (cache) 56ms 2ms 104µs 61ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 22ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 311µs (cache) (cache) 19ms 1ms 57µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.06kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 766B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c91b72da-9cfd-4110 │ -92d0-933b0d39266f",model_name="mnist",model_version=" │ default",} 238.51799999999997 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c91b72da-9cfd- │ 4110-92d0-933b0d39266f",model_name="mnist",model_versi │ on="default",} 61004.298 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c91b72da-9cfd-4110-9 │ 2d0-933b0d39266f",model_name="mnist",model_version="de │ fault",} 2.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 660µs 179µs 227µs 15ms 1ms 67µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 2.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 14.97kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 445ms [min: 19ms, max: 2s, s.d.: 823ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 641µs [min: 179µs, max: 1ms, s.d.: 465µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 439ms [min: 15ms, max: 2s, s.d.: 819ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. Current date and time : 2023-04-15 19:57:25 ------------------------------------------------------------------------------------------ Environment headers ------------------------------------------------------------------------------------------ Torchserve branch: feat/chunked torchserve==0.7.1 torch-model-archiver==0.7.1 Python version: 3.8 (64-bit runtime) Python executable: /Users/lninga/opt/anaconda3/envs/py38/bin/python3 Versions of relevant python libraries: captum==0.6.0 numpy==1.24.2 nvgpu==0.9.0 psutil==5.9.4 pygit2==1.11.1 pylint==2.6.0 pytest==7.2.2 pytest-cov==4.0.0 pytest-mock==3.10.0 requests==2.28.2 requests-toolbelt==0.9.1 torch==1.13.1 torch-model-archiver==0.7.1b20230415 torch-workflow-archiver==0.2.7b20230415 torchaudio==0.13.1 torchserve==0.7.1b20230415 torchserve-nightly==2022.11.14 torchtext==0.14.1 torchvision==0.14.1 transformers==4.25.1 wheel==0.40.0 torch==1.13.1 torchtext==0.14.1 torchvision==0.14.1 torchaudio==0.13.1 Java Version: OS: Mac OSX 13.3.1 (x86_64) GCC version: N/A Clang version: 12.0.0 (clang-1200.0.32.2) CMake version: version 3.19.2 Versions of npm installed packages: markdown-link-check@3.10.3 UNMET PEER DEPENDENCY newman@5.3.2 newman-reporter-html@1.0.5 ## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall . ## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall model-archiver/. ## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall workflow-archiver/. ## Starting generate_mars, mar_config:/Volumes/workplace/python_env/serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## fastrcnn.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## alexnet.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## densenet161.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## mnist.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## resnet-152-batch.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## resnet-18.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## squeezenet1_1.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## vgg16.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## deeplabv3_resnet_101_eager.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## fcn_resnet_101.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## maskrcnn.mar is generated. ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kfv2_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 156ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 32ms 7ms 1ms 612µs 112ms 27ms 5ms 358µs 188ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 51ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 215µs 394µs 31ms 15ms 1ms 87µs 52ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 41ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 986µs 1ms 424µs 30ms 5ms 1ms 64µs 41ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 1982ms time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 399µs (cache) (cache) (cache) 1979ms 1ms 73µs 1983ms ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 6ms time ★ 238B↑ 737B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 470B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-04-15T20:35:55.927Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 31306, │ "gpu": false, │ "gpuUsage": "N/A" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 269µs (cache) (cache) (cache) 3ms 1ms 57µs 6ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 2.3s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 270µs (cache) (cache) (cache) 2.3s 1ms 45µs 2.3s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 341µs (cache) (cache) (cache) 2ms 1ms 46µs 5ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v2/models/mnist/infer 200 OK ★ 21ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 7ms 701µs (cache) (cache) (cache) 17ms 1ms 42µs 27ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v2/models/mnist/explain 200 OK ★ 55ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752831520071, │ -0.0002261287725670648, │ -0.0001273413024972421, │ 0.005648369417700337, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 325µs (cache) (cache) (cache) 53ms 1ms 39µs 57ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 93ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 266µs (cache) (cache) (cache) 90ms 1ms 34µs 93ms ✓ Successful DELETE request ┌─────────────────────────┬────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴────────────────────┴────────────────────┤ │ total run duration: 5s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 99.3kB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 473ms [min: 5ms, max: 2.3s, s.d.: 844ms] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 1ms [min: 215µs, max: 1ms, s.d.: 670µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 451ms [min: 2ms, max: 2.3s, s.d.: 853ms] │ └───────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman explanation_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 4ms 1ms 248µs 2s 7ms 437µs 2.1s ✓ Successful POST request → Model Zoo - Explanations Model POST http://localhost:8080/explanations/mnist 200 OK ★ 77ms time ★ 538B↑ 23.21kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 22.97kB │ [ │ [ │ [ │ -0.00039919446961835236, │ -0.00019002183512235264, │ -0.0008597190403451449, │ -0.0003293672081608754, │ -0.0009114924467526127, │ -0.00017816114351215542, │ -0.0005801028758730413, │ -5.7529225598252194e-05, │ -0.00013036391396408506, │ -3.622336897593853e-05, │ 1.2628194609278315e-05, │ -3.1516328135878544e-05, │ -7.058358081496025e-05, │ -6.872769973604391e-05, │ -7.877429140005835e-05, │ -0.0001330244624280486, │ -3.9255966485147735e-05, │ -0.00022929553875323679, │ 1.713108952478634e-05, │ -0.0003846352858902638, │ -0.0005095514386928224, │ -0.00039880105508864643, │ -0.0008459617489628804, │ -0.00050347912400287, │ -0.0006029825256197108, │ -0.0001768611591953732, │ -0.00015337618628128994, │ -0.0 │ ], │ [ │ -0.000459367041736051, │ -0.0006972018770355594, │ -0.0006564253391229728, │ -0.0006832953398625042, │ -0.0004380070958294335, │ -0.0006188232145845846, │ 8.717674968821532e-06, │ -0.0003930582389242723, │ -0.00010256507980626367, │ -0.00018236528139309367, │ -0.0005491941234231536, │ 4.5198500755516835e-05, │ -0.0004397756033804491, │ -0.00032800121287585753, │ -0.0003846252248742354, │ -0.0007280317477269334, │ -0.0003807700215901187, │ -0.0006797484384283201, │ -0.0009410585867905978, │ -0.0005321376784251184, │ -0.0010959784640141236, │ -0.0009239369515597444, │ -0.0007882725965909251, │ -0.000763552145658654, │ -0.0003335173411764801, │ 5.101228322605105e-05, │ 7.699817024785627e-05, │ -0.0 │ ], │ [ │ -0.0013165055828473268, │ -0.0009519237154533972, │ -0.002100062134240652, │ -0.0007986434565747188, │ -0.0015434154623847722, │ -0.0008677095825202944, │ -0.0006676295763002351, │ -0.0010158373275630642, │ -0.0006892678395626808, │ -0.0010906412181402913, │ -0.0009184818011024204, │ -0.0010186683768814075, │ │ (showing 2.05kB/22.97kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 8ms 1ms 637µs 276µs 72ms 1ms 58µs 85ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 21ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 434µs (cache) (cache) 18ms 1ms 79µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 754B │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c956bdd9-0cbb-4056-b │ 16c-8aac83e7f0b3",model_name="mnist",model_version="de │ fault",} 1.0 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c956bdd9-0cbb-4056 │ -b16c-8aac83e7f0b3",model_name="mnist",model_version=" │ default",} 151.162 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c956bdd9-0cbb- │ 4056-b16c-8aac83e7f0b3",model_name="mnist",model_versi │ on="default",} 51182.37 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 624µs 167µs 223µs 15ms 1ms 63µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 2.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 23.85kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 553ms [min: 19ms, max: 2s, s.d.: 891ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 749µs [min: 167µs, max: 1ms, s.d.: 427µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 547ms [min: 15ms, max: 2s, s.d.: 887ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman management_api_collection Iteration 1/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 41.5s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 5ms 1ms 293µs 41.5s 7ms 444µs 41.5s ✓ Successful request Iteration 2/11 → workflow management request GET http://localhost:8081/workflows 200 OK ★ 9ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B │ { │ "workflows": [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws. │ com/war_files/densenet_wf.war" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 670µs (cache) (cache) 5ms 1ms 79µs 10ms ✓ Successful request Iteration 3/11 → workflow management request GET http://localhost:8081/workflows/densenet 200 OK ★ 6ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B │ [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws.co │ m/war_files/densenet_wf.war", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 50, │ "workflowDag": "{pre_processing=[densenet], densen │ et=[post_processing]}" │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 830µs (cache) (cache) 2ms 1ms 73µs 6ms ✓ Successful request Iteration 4/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 500 Internal Server Error ★ 5ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "code": 500, │ "type": "FileAlreadyExistsException", │ "message": "densenet_wf.war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 2ms 1ms 56µs 6ms ✓ Successful request Iteration 5/11 → workflow management request DELETE http://localhost:8081/workflows/densenet 200 OK ★ 54ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 1ms 229µs 242µs 50ms 1ms 72µs 55ms ✓ Successful request Iteration 6/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war 400 Bad Request ★ 1956ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.s3.amazonaws.com/war_files/does_not_exist. │ war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 229µs (cache) (cache) 1953ms 1ms 46µs 1956ms ✓ Successful request Iteration 7/11 → workflow management request GET http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 14ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 547µs 168µs 223µs 11ms 1ms 44µs 15ms ✓ Successful request Iteration 8/11 → workflow management request DELETE http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 15ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 698µs 178µs 251µs 12ms 1ms 39µs 16ms ✓ Successful request Iteration 9/11 → workflow management request POST http://localhost:8081/workflows?url=malformed_url,? 404 Not Found ★ 15ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found in workflow store: ma │ lformed_url,?" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 583µs 166µs 253µs 12ms 1ms 37µs 16ms ✓ Successful request Iteration 10/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war 500 Internal Server Error ★ 1459ms time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow custom_python_dep has failed to │ register. Failures: [Workflow Node custom_python_dep_ │ _custom_python_dep failed to register. Details: Model │ not found at: custom_python_dep.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 1ms 156µs 219µs 1455ms 1ms 34µs 1460ms ✓ Successful request Iteration 11/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war 500 Internal Server Error ★ 1125ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow loading-memory-error has failed │ to register. Failures: [Workflow Node loading-memory- │ error__loading-memory-error failed to register. Detail │ s: Model not found at: loading-memory-error.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 500µs 141µs 179µs 1121ms 1ms 41µs 1125ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 46.5s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.7kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 4.1s [min: 5ms, max: 41.5s, s.d.: 11.8s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 764µs [min: 141µs, max: 1ms, s.d.: 669µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 4.1s [min: 2ms, max: 41.5s, s.d.: 11.8s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman workflow_inference Iteration 1/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 1m 7.6s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 5ms 1ms 292µs 1m 7.6s 13ms 537µs 1m 7.6s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 200 OK ★ 669ms time ★ 111.24kB↑ 450B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 183B │ { │ "tabby": 0.46661895513534546, │ "tiger_cat": 0.46449047327041626, │ "Egyptian_cat": 0.06614057719707489, │ "lynx": 0.0012924439506605268, │ "plastic_bag": 0.00022909804829396307 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 470µs 631µs 661ms 4ms 78µs 676ms ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 60ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 581µs (cache) (cache) 57ms 2ms 109µs 62ms ✓ Successful DELETE request Iteration 2/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/timeout-retry-test.war 200 OK ★ 58.8s time ★ 328B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 245µs (cache) (cache) 58.8s 2ms 78µs 58.8s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 500 Internal Server Error ★ 12s time ★ 111.24kB↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Error executing pre_processing" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 4ms (cache) (cache) 12s 2ms 65µs 12s ✓ Successful request → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 71ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 377µs (cache) (cache) 68ms 1ms 60µs 72ms ✓ Successful DELETE request Iteration 3/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/dog_breed_wf.war 200 OK ★ 1m 14.1s time ★ 322B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Workflow dog_breed_wf has been registered │ and scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 478µs (cache) (cache) 1m 14.1s 3ms 69µs 1m 14.1s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/dog_breed_wf 200 OK ★ 523ms time ★ 77.75kB↑ 473B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 77.47kB │ (showing 2.05kB/77.47kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 206B │ { │ "Cardigan_welsh_corgi": 0.6281960606575012, │ "Briard": 0.27462849020957947, │ "Icelandic_sheepdog": 0.07453601807355881, │ "Tibetan_mastiff": 0.015545750968158245, │ "Chow_chow": 0.002062038518488407 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 840µs 301µs 322µs 519ms 1ms 52µs 525ms ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/dog_breed_wf 200 OK ★ 36ms time ★ 251B↑ 323B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 57B │ { │ "status": "Workflow \"dog_breed_wf\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 283µs (cache) (cache) 33ms 1ms 49µs 37ms ✓ Successful DELETE request ┌─────────────────────────┬───────────────────────┬──────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼───────────────────────┼──────────────────────┤ │ iterations │ 3 │ 0 │ ├─────────────────────────┼───────────────────────┼──────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼───────────────────────┼──────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼───────────────────────┼──────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼───────────────────────┼──────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴───────────────────────┴──────────────────────┤ │ total run duration: 3m 34.4s │ ├────────────────────────────────────────────────────────────────────────┤ │ total data received: 901B (approx) │ ├────────────────────────────────────────────────────────────────────────┤ │ average response time: 23.7s [min: 36ms, max: 1m 14.1s, s.d.: 30.9s] │ ├────────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 882µs [min: 301µs, max: 1ms, s.d.: 512µs] │ ├────────────────────────────────────────────────────────────────────────┤ │ average first byte time: 23.7s [min: 33ms, max: 1m 14.1s, s.d.: 30.9s] │ └────────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ============================= test session starts ============================== platform darwin -- Python 3.8.13, pytest-7.2.2, pluggy-1.0.0 -- /Users/lninga/opt/anaconda3/envs/py38/bin/python cachedir: .pytest_cache rootdir: /Volumes/workplace/python_env/serve plugins: mock-3.10.0, cov-4.0.0 collecting ... collected 94 items test_example_dcgan.py::test_model_archive_creation PASSED [ 1%] test_example_dcgan.py::test_model_register_unregister PASSED [ 2%] test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [ 3%] test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [ 4%] test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [ 5%] test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [ 6%] test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [ 7%] test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [ 8%] test_example_scriptable_tokenzier.py::test_handler PASSED [ 9%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 10%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string PASSED [ 11%] test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model PASSED [ 12%] test_gRPC_inference_api.py::test_inference_apis PASSED [ 13%] test_gRPC_inference_api.py::test_inference_stream_apis PASSED [ 14%] test_gRPC_management_apis.py::test_management_apis PASSED [ 15%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 17%] test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 18%] test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 19%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 20%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 21%] test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 22%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 23%] test_handler.py::test_huggingface_bert_batch_inference PASSED [ 24%] test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 25%] test_handler.py::test_huggingface_bert_model_parallel_inference PASSED [ 26%] test_metrics.py::test_logs_created PASSED [ 27%] test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 28%] test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 29%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 30%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 31%] test_metrics.py::test_log_location_var_snapshot_disabled PASSED [ 32%] test_metrics.py::test_log_location_var_snapshot_enabled PASSED [ 34%] test_metrics.py::test_async_logging PASSED [ 35%] test_metrics.py::test_async_logging_non_boolean PASSED [ 36%] test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED [ 37%] test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED [ 38%] test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 39%] test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 40%] test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 41%] test_metrics.py::test_collect_system_metrics_when_not_disabled PASSED [ 42%] test_metrics.py::test_disable_system_metrics_using_config_properties PASSED [ 43%] test_metrics.py::test_disable_system_metrics_using_environment_variable PASSED [ 44%] test_metrics_kf.py::test_logs_created PASSED [ 45%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 46%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 47%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 48%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 50%] test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED [ 51%] test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED [ 52%] test_metrics_kf.py::test_async_logging PASSED [ 53%] test_metrics_kf.py::test_async_logging_non_boolean PASSED [ 54%] test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED [ 55%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED [ 56%] test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 57%] test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 58%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 59%] test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 60%] test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 61%] test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 62%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 63%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 64%] test_model_archiver.py::test_model_archiver_without_handler_flag PASSED [ 65%] test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 67%] test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 68%] test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 69%] test_onnx.py::test_convert_to_onnx SKIPPED (ONNX is not installed) [ 70%] test_onnx.py::test_model_packaging_and_start SKIPPED (ONNX is not in...) [ 71%] test_onnx.py::test_model_start SKIPPED (ONNX is not installed) [ 72%] test_onnx.py::test_inference SKIPPED (ONNX is not installed) [ 73%] test_onnx.py::test_stop SKIPPED (ONNX is not installed) [ 74%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 75%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 76%] test_pytorch_profiler.py::test_profiler_arguments_override[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 77%] test_pytorch_profiler.py::test_batch_input[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 78%] test_sm_mme_requirements.py::test_no_model_loaded PASSED [ 79%] test_sm_mme_requirements.py::test_oom_on_model_load SKIPPED (Test to...) [ 80%] test_sm_mme_requirements.py::test_oom_on_invoke SKIPPED (Test to be ...) [ 81%] test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED [ 82%] test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED [ 84%] test_snapshot.py::test_start_from_snapshot PASSED [ 85%] test_snapshot.py::test_start_from_latest PASSED [ 86%] test_snapshot.py::test_start_from_read_only_snapshot PASSED [ 87%] test_snapshot.py::test_no_config_snapshots_cli_option PASSED [ 88%] test_snapshot.py::test_start_from_default PASSED [ 89%] test_snapshot.py::test_start_from_non_existing_snapshot PASSED [ 90%] test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 91%] test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 92%] test_snapshot.py::test_replace_mar_file_with_dummy PASSED [ 93%] test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [ 94%] test_torch_xla.py::TestTorchXLA::test_archive_model_artifacts SKIPPED [ 95%] test_torch_xla.py::TestTorchXLA::test_start_torchserve SKIPPED (PyTo...) [ 96%] test_torch_xla.py::TestTorchXLA::test_server_status SKIPPED (PyTorch...) [ 97%] test_torch_xla.py::TestTorchXLA::test_registered_model SKIPPED (PyTo...) [ 98%] test_torch_xla.py::TestTorchXLA::test_serve_inference SKIPPED (PyTor...) [100%] =============================== warnings summary =============================== ../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:121 /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) ../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870 /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870 ../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870 /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/torch/jit/_recursive.py:262: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/NestedTensorImpl.cpp:177.) return forward_call(*input, **kwargs) test/pytest/test_example_scriptable_tokenzier.py::test_handler /Volumes/workplace/python_env/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. data = F.softmax(data) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========== 77 passed, 17 skipped, 7 warnings in 7890.73s (2:11:30) ============ ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Started regression tests ## Started densenet mar creation ## In directory: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store | Executing command: torch-model-archiver --model-name densenet161_v1 --version 1.1 --model-file /Volumes/workplace/python_env/serve/ts_scripts/../examples/image_classifier/densenet_161/model.py --serialized-file /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store/densenet161-8d451a50.pth --extra-files /Volumes/workplace/python_env/serve/ts_scripts/../examples/image_classifier/index_to_name.json --handler image_classifier --force ## Started regression pytests ## In directory: /Volumes/workplace/python_env/serve/test/pytest | Executing command: python -m pytest -v ./ Removing file : inference_pb2_grpc.py Removing file : inference_pb2.py Removing file : management_pb2.py Removing file : management_pb2_grpc.py ## Deleting model_store_gen_dir: /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen