omninets train result is very low #47
Comments
|
[03/18 09:12:03 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3028 s/iter. Eval: 1.2359 s/iter. Total: 1.5396 s/iter. ETA=0:00:56
100%|██████████| 20/20 [00:07<00:00, 2.72it/s][03/18 09:14:17 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 09:45:01 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3173 s/iter. Eval: 1.0703 s/iter. Total: 1.3884 s/iter. ETA=0:00:51
100%|██████████| 20/20 [00:08<00:00, 2.46it/s][03/18 09:47:13 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 10:18:45 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3187 s/iter. Eval: 0.7140 s/iter. Total: 1.0338 s/iter. ETA=0:00:38
100%|██████████| 20/20 [00:08<00:00, 2.45it/s][03/18 10:20:32 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 10:52:04 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3022 s/iter. Eval: 0.6537 s/iter. Total: 0.9569 s/iter. ETA=0:00:35
100%|██████████| 20/20 [00:08<00:00, 2.39it/s][03/18 10:53:37 tridet]: Evaluation results for kitti_3d_val in csv format:
|
|
[03/18 10:53:55 d2.utils.events]: eta: 1 day, 3:08:42 iter: 10020 total_loss: 2.608 loss_box2d_reg: 0.1859 loss_box3d_depth: 0.6881 loss_box3d_proj_ctr: 0.1245 loss_box3d_quat: 0.188 loss_box3d_size: 0.09417 loss_centerness: 0.6208 loss_cls: 0.1405 loss_conf3d: 0.5876 lr: 0.002 max_mem: 5634M [03/18 11:25:24 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0011 s/iter. Inference: 0.3317 s/iter. Eval: 0.4896 s/iter. Total: 0.8225 s/iter. ETA=0:00:30
100%|██████████| 20/20 [00:08<00:00, 2.43it/s][03/18 11:26:46 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 11:57:43 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3177 s/iter. Eval: 0.3748 s/iter. Total: 0.6935 s/iter. ETA=0:00:25
100%|██████████| 20/20 [00:08<00:00, 2.35it/s][03/18 11:58:56 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 12:30:09 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3160 s/iter. Eval: 0.3866 s/iter. Total: 0.7035 s/iter. ETA=0:00:26
100%|██████████| 20/20 [00:08<00:00, 2.33it/s][03/18 12:31:22 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 13:02:48 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3115 s/iter. Eval: 0.3154 s/iter. Total: 0.6278 s/iter. ETA=0:00:23
100%|██████████| 20/20 [00:08<00:00, 2.32it/s][03/18 13:03:55 tridet]: Evaluation results for kitti_3d_val in csv format:
|
|
[03/18 13:04:13 d2.utils.events]: eta: 9:58:05 iter: 18020 total_loss: 2.398 loss_box2d_reg: 0.1528 loss_box3d_depth: 0.6664 loss_box3d_proj_ctr: 0.08491 loss_box3d_quat: 0.1119 loss_box3d_size: 0.07235 loss_centerness: 0.6144 loss_cls: 0.09708 loss_conf3d: 0.581 lr: 0.002 max_mem: 5634M [03/18 13:35:32 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3236 s/iter. Eval: 0.2405 s/iter. Total: 0.5652 s/iter. ETA=0:00:20
100%|██████████| 20/20 [00:08<00:00, 2.31it/s][03/18 13:36:35 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 14:09:09 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3296 s/iter. Eval: 0.2070 s/iter. Total: 0.5375 s/iter. ETA=0:00:19
100%|██████████| 20/20 [00:09<00:00, 2.21it/s][03/18 14:10:11 tridet]: Evaluation results for kitti_3d_val in csv format:
|
|
[03/18 14:10:30 d2.utils.events]: eta: 3:56:45 iter: 22020 total_loss: 1.756 loss_box2d_reg: 0.1245 loss_box3d_depth: 0.2839 loss_box3d_proj_ctr: 0.06876 loss_box3d_quat: 0.07323 loss_box3d_size: 0.05928 loss_centerness: 0.6114 loss_cls: 0.07886 loss_conf3d: 0.4414 lr: 0.0002 max_mem: 5634M [03/18 14:42:23 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3297 s/iter. Eval: 0.2114 s/iter. Total: 0.5421 s/iter. ETA=0:00:20
100%|██████████| 20/20 [00:09<00:00, 2.20it/s][03/18 14:43:24 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 14:59:41 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0008 s/iter. Inference: 0.3292 s/iter. Eval: 0.1822 s/iter. Total: 0.5123 s/iter. ETA=0:00:18
100%|██████████| 20/20 [00:09<00:00, 2.20it/s][03/18 15:00:41 tridet]: Evaluation results for kitti_3d_val in csv format:
|
|
[03/18 15:00:41 tridet]: Starting inference with test-time augmentation. [03/18 15:00:50 d2.evaluation.evaluator]: Inference done 1/48. Dataloading: 1.1408 s/iter. Inference: 6.7434 s/iter. Eval: 0.5676 s/iter. Total: 8.4560 s/iter. ETA=0:06:37
100%|██████████| 20/20 [00:10<00:00, 1.86it/s][03/18 15:07:25 tridet]: Evaluation results for kitti_3d_val in csv format:
|
Uh oh!
There was an error while loading. Please reload this page.
[03/18 08:06:01 tridet.utils.hydra.callbacks]: Rank of current process: 0. World size: 8
[03/18 08:06:01 tridet.utils.setup]: Working Directory: /home/azuryl/dd3d_test/omni/dd3d/outputs/2023-03-18/08-05-55
[03/18 08:06:01 tridet.utils.setup]: Full config:
{
"WANDB": {
"ENABLED": false,
"DRYRUN": false,
"PROJECT": "dd3d",
"GROUP": null,
"TAGS": [
"kitti-val",
"dla34",
"bn"
]
},
"EVAL_ONLY": false,
"EVAL_ON_START": false,
"ONLY_REGISTER_DATASETS": false,
"OUTPUT_ROOT": "./outputs",
"SYNC_OUTPUT_DIR_S3": {
"ENABLED": false,
"ROOT_IN_S3": "???",
"PERIOD": 1000
},
"DATASET_ROOT": "/hdd1/datasets_left/",
"TMP_DIR": "/tmp/",
"DATASETS": {
"TRAIN": {
"NAME": "kitti_3d_train",
"CANONICAL_BOX3D_SIZES": [
[
1.61876949,
3.89154523,
1.52969237
],
[
0.62806586,
0.82038497,
1.76784787
],
[
0.56898187,
1.77149234,
1.7237099
],
[
1.9134491,
5.15499603,
2.18998422
],
[
2.61168401,
9.22692319,
3.36492722
],
[
0.5390196,
1.08098042,
1.28392158
],
[
2.36044838,
15.56991038,
3.5289238
],
[
1.24489164,
2.51495357,
1.61402478
]
],
"DATASET_MAPPER": "default",
"NUM_CLASSES": 5,
"MEAN_DEPTH_PER_LEVEL": [
32.594,
15.178,
8.424,
5.004,
4.662
],
"STD_DEPTH_PER_LEVEL": [
14.682,
7.139,
4.345,
2.399,
2.587
]
},
"TEST": {
"NAME": "kitti_3d_val",
"NUSC_SAMPLE_AGGREGATE_IN_INFERENCE": false,
"DATASET_MAPPER": "default"
}
},
"FE": {
"FPN": {
"IN_FEATURES": [
"level3",
"level4",
"level5"
],
"OUT_FEATURES": null,
"OUT_CHANNELS": 256,
"NORM": "FrozenBN",
"FUSE_TYPE": "sum"
},
"BUILDER": "build_fcos_dla_fpn_backbone_p67",
"BACKBONE": {
"NAME": "DLA-34",
"OUT_FEATURES": [
"level3",
"level4",
"level5"
],
"NORM": "FrozenBN"
},
"OUT_FEATURES": null
},
"DD3D": {
"IN_FEATURES": null,
"NUM_CLASSES": 5,
"FEATURE_LOCATIONS_OFFSET": "none",
"SIZES_OF_INTEREST": [
64,
128,
256,
512
],
"INFERENCE": {
"DO_NMS": true,
"DO_POSTPROCESS": true,
"DO_BEV_NMS": false,
"BEV_NMS_IOU_THRESH": 0.3,
"NUSC_SAMPLE_AGGREGATE": false
},
"FCOS2D": {
"_VERSION": "v2",
"NORM": "BN",
"NUM_CLS_CONVS": 4,
"NUM_BOX_CONVS": 4,
"USE_DEFORMABLE": false,
"USE_SCALE": true,
"BOX2D_SCALE_INIT_FACTOR": 1.0,
"LOSS": {
"ALPHA": 0.25,
"GAMMA": 2.0,
"LOC_LOSS_TYPE": "giou"
},
"INFERENCE": {
"THRESH_WITH_CTR": true,
"PRE_NMS_THRESH": 0.05,
"PRE_NMS_TOPK": 1000,
"POST_NMS_TOPK": 100,
"NMS_THRESH": 0.75
}
},
"FCOS3D": {
"NORM": "FrozenBN",
"NUM_CONVS": 4,
"USE_DEFORMABLE": false,
"USE_SCALE": true,
"DEPTH_SCALE_INIT_FACTOR": 0.3,
"PROJ_CTR_SCALE_INIT_FACTOR": 1.0,
"PER_LEVEL_PREDICTORS": false,
"SCALE_DEPTH_BY_FOCAL_LENGTHS": true,
"SCALE_DEPTH_BY_FOCAL_LENGTHS_FACTOR": 500.0,
"MEAN_DEPTH_PER_LEVEL": [
32.594,
15.178,
8.424,
5.004,
4.662
],
"STD_DEPTH_PER_LEVEL": [
14.682,
7.139,
4.345,
2.399,
2.587
],
"MIN_DEPTH": 0.1,
"MAX_DEPTH": 80.0,
"CANONICAL_BOX3D_SIZES": [
[
1.61876949,
3.89154523,
1.52969237
],
[
0.62806586,
0.82038497,
1.76784787
],
[
0.56898187,
1.77149234,
1.7237099
],
[
1.9134491,
5.15499603,
2.18998422
],
[
2.61168401,
9.22692319,
3.36492722
],
[
0.5390196,
1.08098042,
1.28392158
],
[
2.36044838,
15.56991038,
3.5289238
],
[
1.24489164,
2.51495357,
1.61402478
]
],
"CLASS_AGNOSTIC_BOX3D": false,
"PREDICT_ALLOCENTRIC_ROT": true,
"PREDICT_DISTANCE": false,
"LOSS": {
"SMOOTH_L1_BETA": 0.05,
"MAX_LOSS_PER_GROUP_DISENT": 20.0,
"CONF_3D_TEMPERATURE": 1.0,
"WEIGHT_BOX3D": 2.0,
"WEIGHT_CONF3D": 1.0
},
"PREPARE_TARGET": {
"CENTER_SAMPLE": true,
"POS_RADIUS": 1.5
}
}
},
"VIS": {
"DATALOADER_ENABLED": true,
"DATALOADER_PERIOD": 1000,
"DATALOADER_MAX_NUM_SAMPLES": 10,
"PREDICTIONS_ENABLED": true,
"PREDICTIONS_MAX_NUM_SAMPLES": 20,
"D2": {
"DATALOADER": {
"ENABLED": true,
"SCALE": 1.0,
"COLOR_MODE": "image"
},
"PREDICTIONS": {
"ENABLED": true,
"SCALE": 1.0,
"COLOR_MODE": "image",
"THRESHOLD": 0.4
}
},
"BOX3D": {
"DATALOADER": {
"ENABLED": true,
"SCALE": 1.0,
"RENDER_LABELS": true
},
"PREDICTIONS": {
"ENABLED": true,
"SCALE": 1.0,
"RENDER_LABELS": true,
"THRESHOLD": 0.5,
"MIN_DEPTH_CENTER": 0.0
}
}
},
"INPUT": {
"FORMAT": "BGR",
"AUG_ENABLED": true,
"RESIZE": {
"ENABLED": true,
"MIN_SIZE_TRAIN": [
288,
304,
320,
336,
352,
368,
384,
400,
416,
448,
480,
512,
544,
576
],
"MIN_SIZE_TRAIN_SAMPLING": "choice",
"MAX_SIZE_TRAIN": 10000,
"MIN_SIZE_TEST": 384,
"MAX_SIZE_TEST": 100000
},
"CROP": {
"ENABLED": false,
"TYPE": "relative_range",
"SIZE": [
0.9,
0.9
]
},
"RANDOM_FLIP": {
"ENABLED": true,
"HORIZONTAL": true,
"VERTICAL": false
},
"COLOR_JITTER": {
"ENABLED": true,
"BRIGHTNESS": [
0.2,
0.2
],
"SATURATION": [
0.2,
0.2
],
"CONTRAST": [
0.2,
0.2
]
}
},
"MODEL": {
"DEVICE": "cuda",
"META_ARCHITECTURE": "DD3D",
"PIXEL_MEAN": [
103.53,
116.28,
123.675
],
"PIXEL_STD": [
57.375,
57.12,
58.395
],
"CKPT": "",
"BOX2D_ON": true,
"BOX3D_ON": true,
"DEPTH_ON": false,
"backbone_with_fpn": null,
"width_mult": 1.0,
"depth_mult": 1.0
},
"DATALOADER": {
"TRAIN": {
"NUM_WORKERS": 12,
"FILTER_EMPTY_ANNOTATIONS": true,
"SAMPLER": "RepeatFactorTrainingSampler",
"REPEAT_THRESHOLD": 0.4,
"ASPECT_RATIO_GROUPING": false
},
"TEST": {
"NUM_WORKERS": 4,
"SAMPLER": "InferenceSampler"
}
},
"SOLVER": {
"IMS_PER_BATCH": 64,
"BASE_LR": 0.002,
"MOMENTUM": 0.9,
"NESTEROV": false,
"WEIGHT_DECAY": 0.0001,
"WEIGHT_DECAY_NORM": 0.0,
"BIAS_LR_FACTOR": 1.0,
"WEIGHT_DECAY_BIAS": 0.0001,
"GAMMA": 0.1,
"LR_SCHEDULER_NAME": "WarmupMultiStepLR",
"STEPS": [
21500,
24000
],
"WARMUP_FACTOR": 0.0001,
"WARMUP_ITERS": 2000,
"WARMUP_METHOD": "linear",
"CLIP_GRADIENTS": {
"ENABLED": false,
"CLIP_TYPE": "value",
"CLIP_VALUE": 1.0,
"NORM_TYPE": 2.0
},
"CHECKPOINT_PERIOD": 2000,
"MIXED_PRECISION_ENABLED": true,
"DDP_FIND_UNUSED_PARAMETERS": false,
"ACCUMULATE_GRAD_BATCHES": 1,
"SYNCBN_USE_LOCAL_WORKERS": false,
"MAX_ITER": 25000
},
"TEST": {
"ENABLED": true,
"EVAL_PERIOD": 2000,
"EVAL_ON_START": false,
"ADDITIONAL_EVAL_STEPS": [],
"IMS_PER_BATCH": 80,
"AUG": {
"ENABLED": true,
"MIN_SIZES": [
320,
384,
448,
512,
576
],
"MAX_SIZE": 100000,
"FLIP": true
}
},
"EVALUATORS": {
"KITTI3D": {
"IOU_THRESHOLDS": [
0.5,
0.7
],
"ONLY_PREPARE_SUBMISSION": false
}
},
"CKPT": "https://tri-ml-public.s3.amazonaws.com/github/dd3d/pretrained/depth_pretrained_omninet-small-3nxjur71.pth"
}
[03/18 08:06:01 tridet.data.datasets.kitti_3d]: KITTI-3D dataset(s): kitti_3d_train, kitti_3d_val
[03/18 08:06:08 tridet.data.build]: Creating D2 dataset: 15 batches on rank 0.
7%|▋ | 1/15 [00:02<00:28, 2.04s/it][2023-03-18 08:06:11,063][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
13%|█▎ | 2/15 [00:02<00:11, 1.10it/s][2023-03-18 08:06:11,115][root][INFO] - Reducer buckets have been rebuilt in this iteration.
100%|██████████| 15/15 [00:03<00:00, 4.94it/s][03/18 08:06:11 tridet.data.build]: Gathering D2 dataset dicts from all GPU workers...
100%|██████████| 15/15 [00:02<00:00, 5.65it/s][03/18 08:06:13 tridet.data.build]: Done (length=3712, took=1.7s).
WARNING [03/18 08:06:13 tridet.utils.coco]: Using previously cached COCO format annotations at '/tmp/kitti_3d_train_coco_format.json'. You need to clear the cache file if your dataset has been modified.
[03/18 08:06:13 tridet.data.datasets.kitti_3d.build]: COCO json file: /tmp/kitti_3d_train_coco_format.json
[03/18 08:06:20 tridet.data.build]: Creating D2 dataset: 15 batches on rank 0.
7%|▋ | 1/15 [00:01<00:27, 1.97s/it][2023-03-18 08:06:23,076][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
13%|█▎ | 2/15 [00:02<00:11, 1.13it/s][2023-03-18 08:06:23,136][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,136][root][INFO] - Reducer buckets have been rebuilt in this iteration.
100%|██████████| 15/15 [00:02<00:00, 5.69it/s][03/18 08:06:23 tridet.data.build]: Gathering D2 dataset dicts from all GPU workers...
100%|██████████| 15/15 [00:02<00:00, 5.67it/s][03/18 08:06:25 tridet.data.build]: Done (length=3769, took=1.5s).
WARNING [03/18 08:06:25 tridet.utils.coco]: Using previously cached COCO format annotations at '/tmp/kitti_3d_val_coco_format.json'. You need to clear the cache file if your dataset has been modified.
[03/18 08:06:25 tridet.data.datasets.kitti_3d.build]: COCO json file: /tmp/kitti_3d_val_coco_format.json
[03/18 08:06:25 tridet]: Registered 2 datasets:
kitti_3d_train
kitti_3d_val
[03/18 08:06:25 tridet.data.dataset_mappers.dataset_mapper]: Augmentations used in training: [ResizeShortestEdge(short_edge_length=[288, 304, 320, 336, 352, 368, 384, 400, 416, 448, 480, 512, 544, 576], max_size=10000, sample_style='choice'), RandomFlip(), RandomBrightness(intensity_min=0.8, intensity_max=1.2), RandomSaturation(intensity_min=0.8, intensity_max=1.2), RandomContrast(intensity_min=0.8, intensity_max=1.2)]
[03/18 08:06:25 d2.data.build]: Removed 0 images with no usable annotations. 3712 images left.
[03/18 08:06:26 d2.data.build]: Distribution of instances among all 5 categories:
2023-03-18 08:06:26,882][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 0
[2023-03-18 08:06:26,884][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 2
[2023-03-18 08:06:26,885][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 1
[2023-03-18 08:06:26,895][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 3
[2023-03-18 08:06:26,900][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 5
[2023-03-18 08:06:26,901][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 4
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 6
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 7
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,912][torch.distributed.distributed_c10d][INFO] - Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,912][torch.distributed.distributed_c10d][INFO] - Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,914][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,916][torch.distributed.distributed_c10d][INFO] - Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,916][torch.distributed.distributed_c10d][INFO] - Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,916][torch.distributed.distributed_c10d][INFO] - Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[03/18 08:06:26 tridet]: Length of train dataset: 3712
[03/18 08:06:26 tridet]: Starting training
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,214][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,214][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,214][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,297][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[03/18 08:06:53 d2.utils.events]: iter: 20 total_loss: 313.7 loss_box2d_reg: 0.9839 loss_box3d_depth: 8.246 loss_box3d_proj_ctr: 0.5431 loss_box3d_quat: 2.739 loss_box3d_size: 1.337 loss_centerness: 0.829 loss_cls: 297.5 loss_conf3d: 1.106 lr: 1.9198e-05 max_mem: 5630M
[03/18 08:07:11 d2.utils.events]: eta: 6:05:01 iter: 40 total_loss: 16.35 loss_box2d_reg: 0.9829 loss_box3d_depth: 7.286 loss_box3d_proj_ctr: 0.4852 loss_box3d_quat: 2.643 loss_box3d_size: 1.003 loss_centerness: 0.7814 loss_cls: 2.374 loss_conf3d: 0.6532 lr: 3.9196e-05 max_mem: 5630M
[03/18 08:07:29 d2.utils.events]: eta: 6:15:22 iter: 60 total_loss: 15.62 loss_box2d_reg: 0.9841 loss_box3d_depth: 7.074 loss_box3d_proj_ctr: 0.4394 loss_box3d_quat: 2.56 loss_box3d_size: 0.8469 loss_centerness: 0.7516 loss_cls: 2.424 loss_conf3d: 0.4763 lr: 5.9194e-05 max_mem: 5630M
[03/18 08:07:45 d2.utils.events]: eta: 5:43:40 iter: 80 total_loss: 14.58 loss_box2d_reg: 0.9868 loss_box3d_depth: 6.905 loss_box3d_proj_ctr: 0.4186 loss_box3d_quat: 2.341 loss_box3d_size: 0.6894 loss_centerness: 0.7354 loss_cls: 2.056 loss_conf3d: 0.434 lr: 7.9192e-05 max_mem: 5630M
[03/18 08:08:02 d2.utils.events]: eta: 5:47:47 iter: 100 total_loss: 13.93 loss_box2d_reg: 0.9874 loss_box3d_depth: 6.831 loss_box3d_proj_ctr: 0.4029 loss_box3d_quat: 2.229 loss_box3d_size: 0.5956 loss_centerness: 0.7245 loss_cls: 1.662 loss_conf3d: 0.412 lr: 9.919e-05 max_mem: 5630M
[03/18 08:08:18 d2.utils.events]: eta: 5:30:20 iter: 120 total_loss: 13.23 loss_box2d_reg: 0.9885 loss_box3d_depth: 6.772 loss_box3d_proj_ctr: 0.4011 loss_box3d_quat: 2.174 loss_box3d_size: 0.5176 loss_centerness: 0.7142 loss_cls: 1.31 loss_conf3d: 0.4019 lr: 0.00011919 max_mem: 5630M
[03/18 08:08:35 d2.utils.events]: eta: 5:53:07 iter: 140 total_loss: 12.69 loss_box2d_reg: 0.9886 loss_box3d_depth: 6.417 loss_box3d_proj_ctr: 0.3833 loss_box3d_quat: 2.155 loss_box3d_size: 0.4419 loss_centerness: 0.7073 loss_cls: 1.047 loss_conf3d: 0.4043 lr: 0.00013919 max_mem: 5630M
[03/18 08:08:52 d2.utils.events]: eta: 5:50:52 iter: 160 total_loss: 12.11 loss_box2d_reg: 0.9847 loss_box3d_depth: 6.238 loss_box3d_proj_ctr: 0.3848 loss_box3d_quat: 2.105 loss_box3d_size: 0.4187 loss_centerness: 0.7041 loss_cls: 0.8634 loss_conf3d: 0.4038 lr: 0.00015918 max_mem: 5630M
[03/18 08:09:08 d2.utils.events]: eta: 5:36:01 iter: 180 total_loss: 11.85 loss_box2d_reg: 0.9781 loss_box3d_depth: 6.133 loss_box3d_proj_ctr: 0.3975 loss_box3d_quat: 2.138 loss_box3d_size: 0.4007 loss_centerness: 0.7013 loss_cls: 0.8141 loss_conf3d: 0.4097 lr: 0.00017918 max_mem: 5630M
[03/18 08:09:25 d2.utils.events]: eta: 5:49:53 iter: 200 total_loss: 11.32 loss_box2d_reg: 0.9691 loss_box3d_depth: 5.581 loss_box3d_proj_ctr: 0.3858 loss_box3d_quat: 2.057 loss_box3d_size: 0.3819 loss_centerness: 0.6958 loss_cls: 0.7868 loss_conf3d: 0.4348 lr: 0.00019918 max_mem: 5630M
[03/18 08:09:42 d2.utils.events]: eta: 5:50:24 iter: 220 total_loss: 10.9 loss_box2d_reg: 0.9443 loss_box3d_depth: 5.266 loss_box3d_proj_ctr: 0.3807 loss_box3d_quat: 2.119 loss_box3d_size: 0.358 loss_centerness: 0.6976 loss_cls: 0.7606 loss_conf3d: 0.436 lr: 0.00021918 max_mem: 5631M
[03/18 08:09:59 d2.utils.events]: eta: 5:36:50 iter: 240 total_loss: 10.65 loss_box2d_reg: 0.8994 loss_box3d_depth: 5.054 loss_box3d_proj_ctr: 0.3808 loss_box3d_quat: 2.041 loss_box3d_size: 0.3253 loss_centerness: 0.6955 loss_cls: 0.7061 loss_conf3d: 0.4434 lr: 0.00023918 max_mem: 5631M
[03/18 08:10:16 d2.utils.events]: eta: 6:03:12 iter: 260 total_loss: 10.58 loss_box2d_reg: 0.8071 loss_box3d_depth: 5.113 loss_box3d_proj_ctr: 0.3784 loss_box3d_quat: 2.126 loss_box3d_size: 0.3223 loss_centerness: 0.6966 loss_cls: 0.6927 loss_conf3d: 0.4359 lr: 0.00025917 max_mem: 5631M
[03/18 08:10:34 d2.utils.events]: eta: 6:12:29 iter: 280 total_loss: 9.967 loss_box2d_reg: 0.7079 loss_box3d_depth: 4.762 loss_box3d_proj_ctr: 0.3807 loss_box3d_quat: 2.071 loss_box3d_size: 0.3053 loss_centerness: 0.697 loss_cls: 0.6697 loss_conf3d: 0.4572 lr: 0.00027917 max_mem: 5631M
[03/18 08:10:52 d2.utils.events]: eta: 6:00:41 iter: 300 total_loss: 9.587 loss_box2d_reg: 0.6188 loss_box3d_depth: 4.493 loss_box3d_proj_ctr: 0.3809 loss_box3d_quat: 2.019 loss_box3d_size: 0.2915 loss_centerness: 0.6956 loss_cls: 0.6371 loss_conf3d: 0.4666 lr: 0.00029917 max_mem: 5631M
[03/18 08:11:09 d2.utils.events]: eta: 6:02:27 iter: 320 total_loss: 9.218 loss_box2d_reg: 0.5775 loss_box3d_depth: 4.211 loss_box3d_proj_ctr: 0.3782 loss_box3d_quat: 2.049 loss_box3d_size: 0.2695 loss_centerness: 0.6879 loss_cls: 0.6051 loss_conf3d: 0.4795 lr: 0.00031917 max_mem: 5631M
[03/18 08:11:26 d2.utils.events]: eta: 5:46:55 iter: 340 total_loss: 9.233 loss_box2d_reg: 0.5667 loss_box3d_depth: 4.277 loss_box3d_proj_ctr: 0.3803 loss_box3d_quat: 1.974 loss_box3d_size: 0.2613 loss_centerness: 0.6883 loss_cls: 0.5957 loss_conf3d: 0.4734 lr: 0.00033917 max_mem: 5631M
[03/18 08:11:42 d2.utils.events]: eta: 5:31:08 iter: 360 total_loss: 8.959 loss_box2d_reg: 0.5531 loss_box3d_depth: 3.969 loss_box3d_proj_ctr: 0.377 loss_box3d_quat: 1.966 loss_box3d_size: 0.2622 loss_centerness: 0.683 loss_cls: 0.5763 loss_conf3d: 0.4865 lr: 0.00035916 max_mem: 5631M
[03/18 08:11:59 d2.utils.events]: eta: 5:48:45 iter: 380 total_loss: 8.931 loss_box2d_reg: 0.5438 loss_box3d_depth: 3.986 loss_box3d_proj_ctr: 0.3742 loss_box3d_quat: 2.071 loss_box3d_size: 0.2466 loss_centerness: 0.6847 8000 loss_cls: 0.5473 loss_conf3d: 0.4756 lr: 0.00037916 max_mem: 5631M
[03/18 08:12:16 d2.utils.events]: eta: 5:44:59 iter: 400 total_loss: 8.547 loss_box2d_reg: 0.53 loss_box3d_depth: 3.688 loss_box3d_proj_ctr: 0.3782 loss_box3d_quat: 2.032 loss_box3d_size: 0.2267 loss_centerness: 0.6795 loss_cls: 0.5292 loss_conf3d: 0.4964 lr: 0.00039916 max_mem: 5631M
[03/18 08:12:33 d2.utils.events]: eta: 5:41:55 iter: 420 total_loss: 8.449 loss_box2d_reg: 0.5203 loss_box3d_depth: 3.59 loss_box3d_proj_ctr: 0.3694 loss_box3d_quat: 2.06 loss_box3d_size: 0.2328 loss_centerness: 0.6779 loss_cls: 0.5298 loss_conf3d: 0.5062 lr: 0.00041916 max_mem: 5631M
[03/18 08:12:50 d2.utils.events]: eta: 5:58:25 iter: 440 total_loss: 8.701 loss_box2d_reg: 0.5144 loss_box3d_depth: 3.845 loss_box3d_proj_ctr: 0.3692 loss_box3d_quat: 2.052 loss_box3d_size: 0.2305 loss_centerness: 0.6765 loss_cls: 0.5357 loss_conf3d: 0.496 lr: 0.00043916 max_mem: 5631M
[03/18 08:13:08 d2.utils.events]: eta: 6:03:39 iter: 460 total_loss: 8.406 loss_box2d_reg: 0.5114 loss_box3d_depth: 3.646 loss_box3d_proj_ctr: 0.3705 loss_box3d_quat: 2.024 loss_box3d_size: 0.2223 loss_centerness: 0.6749 loss_cls: 0.5159 loss_conf3d: 0.5056 lr: 0.00045915 max_mem: 5631M
[03/18 08:13:25 d2.utils.events]: eta: 5:35:23 iter: 480 total_loss: 8.059 loss_box2d_reg: 0.495 loss_box3d_depth: 3.274 loss_box3d_proj_ctr: 0.3645 loss_box3d_quat: 1.989 loss_box3d_size: 0.2091 loss_centerness: 0.673 loss_cls: 0.4985 loss_conf3d: 0.5236 lr: 0.00047915 max_mem: 5631M
[03/18 08:13:43 d2.utils.events]: eta: 6:09:42 iter: 500 total_loss: 8.048 loss_box2d_reg: 0.4968 loss_box3d_depth: 3.301 loss_box3d_proj_ctr: 0.3631 loss_box3d_quat: 2.007 loss_box3d_size: 0.2059 loss_centerness: 0.6718 loss_cls: 0.4829 loss_conf3d: 0.5226 lr: 0.00049915 max_mem: 5631M
[03/18 08:14:02 d2.utils.events]: eta: 6:22:45 iter: 520 total_loss: 7.996 loss_box2d_reg: 0.4874 loss_box3d_depth: 3.196 loss_box3d_proj_ctr: 0.3659 loss_box3d_quat: 2.018 loss_box3d_size: 0.2089 loss_centerness: 0.6704 loss_cls: 0.482 loss_conf3d: 0.5192 lr: 0.00051915 max_mem: 5631M
[03/18 08:14:19 d2.utils.events]: eta: 5:52:08 iter: 540 total_loss: 8.394 loss_box2d_reg: 0.4856 loss_box3d_depth: 3.628 loss_box3d_proj_ctr: 0.3581 loss_box3d_quat: 2.013 loss_box3d_size: 0.2012 loss_centerness: 0.6691 loss_cls: 0.4796 loss_conf3d: 0.5054 lr: 0.00053915 max_mem: 5631M
[03/18 08:14:36 d2.utils.events]: eta: 5:56:55 iter: 560 total_loss: 7.878 loss_box2d_reg: 0.4753 loss_box3d_depth: 3.151 loss_box3d_proj_ctr: 0.3652 loss_box3d_quat: 2.065 loss_box3d_size: 0.1987 loss_centerness: 0.6684 loss_cls: 0.4594 loss_conf3d: 0.5235 lr: 0.00055914 max_mem: 5631M
[03/18 08:14:54 d2.utils.events]: eta: 5:59:40 iter: 580 total_loss: 7.752 loss_box2d_reg: 0.4706 loss_box3d_depth: 3.02 loss_box3d_proj_ctr: 0.3588 loss_box3d_quat: 2.003 loss_box3d_size: 0.1921 loss_centerness: 0.666 loss_cls: 0.4479 loss_conf3d: 0.5276 lr: 0.00057914 max_mem: 5631M
[03/18 08:15:11 d2.utils.events]: eta: 5:50:34 iter: 600 total_loss: 7.661 loss_box2d_reg: 0.4639 loss_box3d_depth: 2.968 loss_box3d_proj_ctr: 0.3508 loss_box3d_quat: 1.98 loss_box3d_size: 0.198 loss_centerness: 0.6653 loss_cls: 0.4515 loss_conf3d: 0.5431 lr: 0.00059914 max_mem: 5631M
[03/18 08:15:29 d2.utils.events]: eta: 6:02:12 iter: 620 total_loss: 7.837 loss_box2d_reg: 0.4569 loss_box3d_depth: 3.175 loss_box3d_proj_ctr: 0.3568 loss_box3d_quat: 1.978 loss_box3d_size: 0.1946 loss_centerness: 0.6636 loss_cls: 0.4433 loss_conf3d: 0.521 lr: 0.00061914 max_mem: 5631M
[03/18 08:15:47 d2.utils.events]: eta: 6:07:58 iter: 640 total_loss: 7.4 loss_box2d_reg: 0.4518 loss_box3d_depth: 2.776 loss_box3d_proj_ctr: 0.3484 loss_box3d_quat: 2.027 loss_box3d_size: 0.1891 loss_centerness: 0.6632 loss_cls: 0.4388 loss_conf3d: 0.5476 lr: 0.00063914 max_mem: 5631M
[03/18 08:16:05 d2.utils.events]: eta: 6:00:15 iter: 660 total_loss: 7.444 loss_box2d_reg: 0.4427 loss_box3d_depth: 2.746 loss_box3d_proj_ctr: 0.3471 loss_box3d_quat: 2.013 loss_box3d_size: 0.1857 loss_centerness: 0.6616 loss_cls: 0.4334 loss_conf3d: 0.5383 lr: 0.00065913 max_mem: 5631M
[03/18 08:16:23 d2.utils.events]: eta: 6:05:13 iter: 680 total_loss: 7.34 loss_box2d_reg: 0.4486 loss_box3d_depth: 2.729 loss_box3d_proj_ctr: 0.3551 loss_box3d_quat: 2.051 loss_box3d_size: 0.1861 loss_centerness: 0.6625 loss_cls: 0.4228 loss_conf3d: 0.5446 lr: 0.00067913 max_mem: 5631M
[03/18 08:16:41 d2.utils.events]: eta: 6:12:37 iter: 700 total_loss: 7.324 loss_box2d_reg: 0.4437 loss_box3d_depth: 2.778 loss_box3d_proj_ctr: 0.3369 loss_box3d_quat: 1.895 loss_box3d_size: 0.1822 loss_centerness: 0.6604 loss_cls: 0.421 loss_conf3d: 0.5549 lr: 0.00069913 max_mem: 5631M
[03/18 08:16:59 d2.utils.events]: eta: 5:53:25 iter: 720 total_loss: 7.247 loss_box2d_reg: 0.4349 loss_box3d_depth: 2.665 loss_box3d_proj_ctr: 0.3395 loss_box3d_quat: 2.071 loss_box3d_size: 0.1832 loss_centerness: 0.66 loss_cls: 0.4182 loss_conf3d: 0.5483 lr: 0.00071913 max_mem: 5631M
[03/18 08:17:18 d2.utils.events]: eta: 6:25:09 iter: 740 total_loss: 7.225 loss_box2d_reg: 0.427 loss_box3d_depth: 2.573 loss_box3d_proj_ctr: 0.3331 loss_box3d_quat: 2.058 loss_box3d_size: 0.1827 loss_centerness: 0.6577 loss_cls: 0.4156 loss_conf3d: 0.5472 lr: 0.00073913 max_mem: 5631M
[03/18 08:17:37 d2.utils.events]: eta: 6:18:08 iter: 760 total_loss: 7.479 loss_box2d_reg: 0.4307 loss_box3d_depth: 2.97 loss_box3d_proj_ctr: 0.3494 loss_box3d_quat: 2.002 loss_box3d_size: 0.1835 loss_centerness: 0.6579 loss_cls: 0.4124 loss_conf3d: 0.539 lr: 0.00075912 max_mem: 5631M
[03/18 08:17:55 d2.utils.events]: eta: 6:13:52 iter: 780 total_loss: 7.41 loss_box2d_reg: 0.4249 loss_box3d_depth: 2.805 loss_box3d_proj_ctr: 0.3338 loss_box3d_quat: 2.003 loss_box3d_size: 0.1762 loss_centerness: 0.6576 loss_cls: 0.4098 loss_conf3d: 0.5387 lr: 0.00077912 max_mem: 5631M
[03/18 08:18:14 d2.utils.events]: eta: 6:14:02 iter: 800 total_loss: 7.089 loss_box2d_reg: 0.4275 loss_box3d_depth: 2.618 loss_box3d_proj_ctr: 0.3316 loss_box3d_quat: 2.004 loss_box3d_size: 0.1814 loss_centerness: 0.6576 loss_cls: 0.4038 loss_conf3d: 0.5579 lr: 0.00079912 max_mem: 5632M
[03/18 08:18:32 d2.utils.events]: eta: 6:06:57 iter: 820 total_loss: 7.155 loss_box2d_reg: 0.4229 loss_box3d_depth: 2.65 loss_box3d_proj_ctr: 0.3232 loss_box3d_quat: 2.034 loss_box3d_size: 0.1791 loss_centerness: 0.6558 loss_cls: 0.4 loss_conf3d: 0.5496 lr: 0.00081912 max_mem: 5632M
[03/18 08:18:49 d2.utils.events]: eta: 5:45:03 iter: 840 total_loss: 6.939 loss_box2d_reg: 0.4146 loss_box3d_depth: 2.444 loss_box3d_proj_ctr: 0.3215 loss_box3d_quat: 2.012 loss_box3d_size: 0.1754 loss_centerness: 0.6562 loss_cls: 0.3967 loss_conf3d: 0.5615 lr: 0.00083912 max_mem: 5632M
[03/18 08:19:07 d2.utils.events]: eta: 6:01:24 iter: 860 total_loss: 7.042 loss_box2d_reg: 0.4196 loss_box3d_depth: 2.52 loss_box3d_proj_ctr: 0.3196 loss_box3d_quat: 1.966 loss_box3d_size: 0.1746 loss_centerness: 0.6566 loss_cls: 0.3916 loss_conf3d: 0.5585 lr: 0.00085911 max_mem: 5632M
[03/18 08:19:25 d2.utils.events]: eta: 6:08:57 iter: 880 total_loss: 7.178 loss_box2d_reg: 0.4112 loss_box3d_depth: 2.596 loss_box3d_proj_ctr: 0.3241 loss_box3d_quat: 1.964 loss_box3d_size: 0.1747 loss_centerness: 0.6557 loss_cls: 0.389 loss_conf3d: 0.5442 lr: 0.00087911 max_mem: 5632M
[03/18 08:19:43 d2.utils.events]: eta: 5:47:11 iter: 900 total_loss: 6.987 loss_box2d_reg: 0.4133 loss_box3d_depth: 2.505 loss_box3d_proj_ctr: 0.3177 loss_box3d_quat: 1.967 loss_box3d_size: 0.165 loss_centerness: 0.6551 loss_cls: 0.3895 loss_conf3d: 0.5635 lr: 0.00089911 max_mem: 5632M
[03/18 08:20:01 d2.utils.events]: eta: 6:12:12 iter: 920 total_loss: 6.741 loss_box2d_reg: 0.399 loss_box3d_depth: 2.223 loss_box3d_proj_ctr: 0.3157 loss_box3d_quat: 2.062 loss_box3d_size: 0.1692 loss_centerness: 0.6537 loss_cls: 0.3846 loss_conf3d: 0.572 lr: 0.00091911 max_mem: 5632M
[03/18 08:20:19 d2.utils.events]: eta: 6:05:16 iter: 940 total_loss: 6.783 loss_box2d_reg: 0.4011 loss_box3d_depth: 2.258 loss_box3d_proj_ctr: 0.315 loss_box3d_quat: 1.948 loss_box3d_size: 0.1725 loss_centerness: 0.6535 loss_cls: 0.3873 loss_conf3d: 0.5749 lr: 0.00093911 max_mem: 5632M
[03/18 08:20:37 d2.utils.events]: eta: 5:46:51 iter: 960 total_loss: 6.665 loss_box2d_reg: 0.3935 loss_box3d_depth: 2.254 loss_box3d_proj_ctr: 0.3167 loss_box3d_quat: 1.942 loss_box3d_size: 0.1703 loss_centerness: 0.6518 loss_cls: 0.3809 loss_conf3d: 0.5796 lr: 0.0009591 max_mem: 5632M
[03/18 08:20:56 d2.utils.events]: eta: 6:30:48 iter: 980 total_loss: 6.87 loss_box2d_reg: 0.3943 loss_box3d_depth: 2.313 loss_box3d_proj_ctr: 0.3179 loss_box3d_quat: 1.956 loss_box3d_size: 0.1642 loss_centerness: 0.6527 loss_cls: 0.38 loss_conf3d: 0.5673 lr: 0.0009791 max_mem: 5632M
[03/18 08:21:15 d2.utils.events]: eta: 6:18:15 iter: 1000 total_loss: 6.81 loss_box2d_reg: 0.3928 loss_box3d_depth: 2.418 loss_box3d_proj_ctr: 0.3128 loss_box3d_quat: 1.942 loss_box3d_size: 0.1678 loss_centerness: 0.6514 loss_cls: 0.3743 loss_conf3d: 0.5641 lr: 0.0009991 max_mem: 5632M
[03/18 08:21:38 d2.utils.events]: eta: 7:39:37 iter: 1020 total_loss: 6.905 loss_box2d_reg: 0.3801 loss_box3d_depth: 2.482 loss_box3d_proj_ctr: 0.3116 loss_box3d_quat: 1.942 loss_box3d_size: 0.1705 loss_centerness: 0.6507 loss_cls: 0.3672 loss_conf3d: 0.5583 lr: 0.0010191 max_mem: 5632M
[03/18 08:21:57 d2.utils.events]: eta: 6:12:08 iter: 1040 total_loss: 6.639 loss_box2d_reg: 0.3866 loss_box3d_depth: 2.269 loss_box3d_proj_ctr: 0.3061 loss_box3d_quat: 1.967 loss_box3d_size: 0.164 loss_centerness: 0.6509 loss_cls: 0.3658 loss_conf3d: 0.5779 lr: 0.0010391 max_mem: 5632M
[03/18 08:22:15 d2.utils.events]: eta: 6:11:48 iter: 1060 total_loss: 6.67 loss_box2d_reg: 0.3877 loss_box3d_depth: 2.281 loss_box3d_proj_ctr: 0.305 loss_box3d_quat: 1.972 loss_box3d_size: 0.1642 loss_centerness: 0.651 loss_cls: 0.3669 loss_conf3d: 0.5748 lr: 0.0010591 max_mem: 5632M
[03/18 08:22:33 d2.utils.events]: eta: 5:52:40 iter: 1080 total_loss: 6.735 loss_box2d_reg: 0.3809 loss_box3d_depth: 2.261 loss_box3d_proj_ctr: 0.3159 loss_box3d_quat: 1.955 loss_box3d_size: 0.1634 loss_centerness: 0.6518 loss_cls: 0.3699 loss_conf3d: 0.5751 lr: 0.0010791 max_mem: 5632M
[03/18 08:22:51 d2.utils.events]: eta: 5:57:27 iter: 1100 total_loss: 6.578 loss_box2d_reg: 0.3794 loss_box3d_depth: 2.198 loss_box3d_proj_ctr: 0.306 loss_box3d_quat: 1.953 loss_box3d_size: 0.1629 loss_centerness: 0.6504 loss_cls: 0.3662 loss_conf3d: 0.5751 lr: 0.0010991 max_mem: 5632M
[03/18 08:23:09 d2.utils.events]: eta: 5:59:44 iter: 1120 total_loss: 6.641 loss_box2d_reg: 0.3826 loss_box3d_depth: 2.281 loss_box3d_proj_ctr: 0.3073 loss_box3d_quat: 1.879 loss_box3d_size: 0.156 loss_centerness: 0.6519 loss_cls: 0.3631 loss_conf3d: 0.5691 lr: 0.0011191 max_mem: 5632M
[03/18 08:23:27 d2.utils.events]: eta: 6:01:38 iter: 1140 total_loss: 6.661 loss_box2d_reg: 0.3768 loss_box3d_depth: 2.177 loss_box3d_proj_ctr: 0.3096 loss_box3d_quat: 1.997 loss_box3d_size: 0.1643 loss_centerness: 0.6498 loss_cls: 0.3563 loss_conf3d: 0.5694 lr: 0.0011391 max_mem: 5632M
[03/18 08:23:46 d2.utils.events]: eta: 6:10:59 iter: 1160 total_loss: 6.691 loss_box2d_reg: 0.3709 loss_box3d_depth: 2.344 loss_box3d_proj_ctr: 0.3047 loss_box3d_quat: 1.934 loss_box3d_size: 0.1581 loss_centerness: 0.6487 loss_cls: 0.3619 loss_conf3d: 0.5679 lr: 0.0011591 max_mem: 5632M
[03/18 08:24:04 d2.utils.events]: eta: 6:04:55 iter: 1180 total_loss: 7.015 loss_box2d_reg: 0.3737 loss_box3d_depth: 2.619 loss_box3d_proj_ctr: 0.3101 loss_box3d_quat: 1.944 loss_box3d_size: 0.1604 loss_centerness: 0.6485 loss_cls: 0.3585 loss_conf3d: 0.5463 lr: 0.0011791 max_mem: 5632M
[03/18 08:24:23 d2.utils.events]: eta: 6:05:43 iter: 1200 total_loss: 6.973 loss_box2d_reg: 0.3769 loss_box3d_depth: 2.603 loss_box3d_proj_ctr: 0.3047 loss_box3d_quat: 1.943 loss_box3d_size: 0.1655 loss_centerness: 0.6503 loss_cls: 0.3687 loss_conf3d: 0.5576 lr: 0.0011991 max_mem: 5632M
[03/18 08:24:42 d2.utils.events]: eta: 6:18:14 iter: 1220 total_loss: 6.539 loss_box2d_reg: 0.375 loss_box3d_depth: 2.191 loss_box3d_proj_ctr: 0.3045 loss_box3d_quat: 1.939 loss_box3d_size: 0.1591 loss_centerness: 0.651 loss_cls: 0.3683 loss_conf3d: 0.5825 lr: 0.0012191 max_mem: 5632M
[03/18 08:25:02 d2.utils.events]: eta: 6:28:30 iter: 1240 total_loss: 6.376 loss_box2d_reg: 0.3713 loss_box3d_depth: 2.121 loss_box3d_proj_ctr: 0.2918 loss_box3d_quat: 1.792 loss_box3d_size: 0.1579 loss_centerness: 0.6496 loss_cls: 0.3558 loss_conf3d: 0.6053 lr: 0.0012391 max_mem: 5632M
[03/18 08:25:20 d2.utils.events]: eta: 6:05:09 iter: 1260 total_loss: 6.324 loss_box2d_reg: 0.3733 loss_box3d_depth: 2.279 loss_box3d_proj_ctr: 0.3035 loss_box3d_quat: 1.608 loss_box3d_size: 0.1526 loss_centerness: 0.6486 loss_cls: 0.3618 loss_conf3d: 0.6009 lr: 0.0012591 max_mem: 5632M
[03/18 08:25:39 d2.utils.events]: eta: 6:15:00 iter: 1280 total_loss: 6.361 loss_box2d_reg: 0.3683 loss_box3d_depth: 2.268 loss_box3d_proj_ctr: 0.31 loss_box3d_quat: 1.615 loss_box3d_size: 0.1639 loss_centerness: 0.6491 loss_cls: 0.3582 loss_conf3d: 0.6014 lr: 0.0012791 max_mem: 5632M
[03/18 08:25:58 d2.utils.events]: eta: 6:07:48 iter: 1300 total_loss: 6.099 loss_box2d_reg: 0.3677 loss_box3d_depth: 2.089 loss_box3d_proj_ctr: 0.3075 loss_box3d_quat: 1.525 loss_box3d_size: 0.165 loss_centerness: 0.6499 loss_cls: 0.3592 loss_conf3d: 0.6165 lr: 0.0012991 max_mem: 5632M
[03/18 08:26:16 d2.utils.events]: eta: 5:54:22 iter: 1320 total_loss: 6.157 loss_box2d_reg: 0.3632 loss_box3d_depth: 2.209 loss_box3d_proj_ctr: 0.3017 loss_box3d_quat: 1.495 loss_box3d_size: 0.1633 loss_centerness: 0.6475 loss_cls: 0.3518 loss_conf3d: 0.614 lr: 0.0013191 max_mem: 5632M
[03/18 08:26:35 d2.utils.events]: eta: 6:20:50 iter: 1340 total_loss: 6.085 loss_box2d_reg: 0.3638 loss_box3d_depth: 2.133 loss_box3d_proj_ctr: 0.3024 loss_box3d_quat: 1.456 loss_box3d_size: 0.1624 loss_centerness: 0.6482 loss_cls: 0.3557 loss_conf3d: 0.6216 lr: 0.0013391 max_mem: 5632M
[03/18 08:26:54 d2.utils.events]: eta: 6:10:26 iter: 1360 total_loss: 6.072 loss_box2d_reg: 0.368 loss_box3d_depth: 2.166 loss_box3d_proj_ctr: 0.3054 loss_box3d_quat: 1.49 loss_box3d_size: 0.1599 loss_centerness: 0.6474 loss_cls: 0.3537 loss_conf3d: 0.6147 lr: 0.0013591 max_mem: 5632M
[03/18 08:27:12 d2.utils.events]: eta: 6:05:03 iter: 1380 total_loss: 5.727 loss_box2d_reg: 0.3611 loss_box3d_depth: 1.988 loss_box3d_proj_ctr: 0.3012 loss_box3d_quat: 1.322 loss_box3d_size: 0.156 loss_centerness: 0.6484 loss_cls: 0.3463 loss_conf3d: 0.6342 lr: 0.0013791 max_mem: 5632M
[03/18 08:27:31 d2.utils.events]: eta: 6:09:04 iter: 1400 total_loss: 5.808 loss_box2d_reg: 0.3535 loss_box3d_depth: 2.093 loss_box3d_proj_ctr: 0.3004 loss_box3d_quat: 1.284 loss_box3d_size: 0.1502 loss_centerness: 0.6469 loss_cls: 0.3494 loss_conf3d: 0.6318 lr: 0.0013991 max_mem: 5632M
[03/18 08:27:50 d2.utils.events]: eta: 6:12:05 iter: 1420 total_loss: 5.784 loss_box2d_reg: 0.3496 loss_box3d_depth: 2.005 loss_box3d_proj_ctr: 0.2989 loss_box3d_quat: 1.343 loss_box3d_size: 0.1568 loss_centerness: 0.6455 loss_cls: 0.3433 loss_conf3d: 0.6302 lr: 0.0014191 max_mem: 5632M
[03/18 08:28:09 d2.utils.events]: eta: 6:04:13 iter: 1440 total_loss: 5.8 loss_box2d_reg: 0.3531 loss_box3d_depth: 2.089 loss_box3d_proj_ctr: 0.2929 loss_box3d_quat: 1.318 loss_box3d_size: 0.1598 loss_centerness: 0.6463 loss_cls: 0.3447 loss_conf3d: 0.6293 lr: 0.0014391 max_mem: 5632M
[03/18 08:28:28 d2.utils.events]: eta: 6:29:27 iter: 1460 total_loss: 5.891 loss_box2d_reg: 0.3608 loss_box3d_depth: 2.124 loss_box3d_proj_ctr: 0.292 loss_box3d_quat: 1.325 loss_box3d_size: 0.1578 loss_centerness: 0.6478 loss_cls: 0.3528 loss_conf3d: 0.6313 lr: 0.0014591 max_mem: 5632M
[03/18 08:28:48 d2.utils.events]: eta: 6:21:46 iter: 1480 total_loss: 5.804 loss_box2d_reg: 0.3537 loss_box3d_depth: 2.145 loss_box3d_proj_ctr: 0.3021 loss_box3d_quat: 1.24 loss_box3d_size: 0.1523 loss_centerness: 0.6464 loss_cls: 0.3407 loss_conf3d: 0.6331 lr: 0.0014791 max_mem: 5632M
[03/18 08:29:07 d2.utils.events]: eta: 6:08:08 iter: 1500 total_loss: 6.109 loss_box2d_reg: 0.3569 loss_box3d_depth: 2.405 loss_box3d_proj_ctr: 0.301 loss_box3d_quat: 1.284 loss_box3d_size: 0.1531 loss_centerness: 0.6459 loss_cls: 0.3489 loss_conf3d: 0.6149 lr: 0.0014991 max_mem: 5632M
[03/18 08:29:25 d2.utils.events]: eta: 6:05:07 iter: 1520 total_loss: 5.78 loss_box2d_reg: 0.3539 loss_box3d_depth: 2.094 loss_box3d_proj_ctr: 0.2956 loss_box3d_quat: 1.225 loss_box3d_size: 0.1588 loss_centerness: 0.6471 loss_cls: 0.3409 loss_conf3d: 0.6292 lr: 0.001519 max_mem: 5632M
[03/18 08:29:44 d2.utils.events]: eta: 5:57:01 iter: 1540 total_loss: 5.629 loss_box2d_reg: 0.3464 loss_box3d_depth: 2.034 loss_box3d_proj_ctr: 0.2877 loss_box3d_quat: 1.189 loss_box3d_size: 0.1499 loss_centerness: 0.6435 loss_cls: 0.3288 loss_conf3d: 0.6345 lr: 0.001539 max_mem: 5632M
[03/18 08:30:01 d2.utils.events]: eta: 5:48:09 iter: 1560 total_loss: 5.533 loss_box2d_reg: 0.3415 loss_box3d_depth: 1.951 loss_box3d_proj_ctr: 0.2918 loss_box3d_quat: 1.182 loss_box3d_size: 0.1534 loss_centerness: 0.6448 loss_cls: 0.3316 loss_conf3d: 0.637 lr: 0.001559 max_mem: 5632M
[03/18 08:30:21 d2.utils.events]: eta: 6:20:16 iter: 1580 total_loss: 6.106 loss_box2d_reg: 0.351 loss_box3d_depth: 2.564 loss_box3d_proj_ctr: 0.2969 loss_box3d_quat: 1.215 loss_box3d_size: 0.1525 loss_centerness: 0.6448 loss_cls: 0.3347 loss_conf3d: 0.6149 lr: 0.001579 max_mem: 5632M
[03/18 08:30:39 d2.utils.events]: eta: 6:00:58 iter: 1600 total_loss: 5.755 loss_box2d_reg: 0.3522 loss_box3d_depth: 2.135 loss_box3d_proj_ctr: 0.2831 loss_box3d_quat: 1.191 loss_box3d_size: 0.1524 loss_centerness: 0.646 loss_cls: 0.3369 loss_conf3d: 0.6341 lr: 0.001599 max_mem: 5632M
[03/18 08:30:57 d2.utils.events]: eta: 5:52:18 iter: 1620 total_loss: 5.558 loss_box2d_reg: 0.3479 loss_box3d_depth: 2.041 loss_box3d_proj_ctr: 0.2937 loss_box3d_quat: 1.187 loss_box3d_size: 0.1515 loss_centerness: 0.6454 loss_cls: 0.3347 loss_conf3d: 0.6389 lr: 0.001619 max_mem: 5632M
[03/18 08:31:16 d2.utils.events]: eta: 5:58:20 iter: 1640 total_loss: 5.535 loss_box2d_reg: 0.3432 loss_box3d_depth: 1.971 loss_box3d_proj_ctr: 0.2949 loss_box3d_quat: 1.18 loss_box3d_size: 0.1521 loss_centerness: 0.6447 loss_cls: 0.3311 loss_conf3d: 0.6382 lr: 0.001639 max_mem: 5632M
[03/18 08:31:35 d2.utils.events]: eta: 6:13:50 iter: 1660 total_loss: 5.641 loss_box2d_reg: 0.3409 loss_box3d_depth: 2.094 loss_box3d_proj_ctr: 0.2829 loss_box3d_quat: 1.119 loss_box3d_size: 0.1496 loss_centerness: 0.6433 loss_cls: 0.3285 loss_conf3d: 0.6339 lr: 0.001659 max_mem: 5632M
[03/18 08:31:53 d2.utils.events]: eta: 5:48:47 iter: 1680 total_loss: 5.577 loss_box2d_reg: 0.3472 loss_box3d_depth: 2.003 loss_box3d_proj_ctr: 0.2884 loss_box3d_quat: 1.109 loss_box3d_size: 0.1493 loss_centerness: 0.6456 loss_cls: 0.3313 loss_conf3d: 0.6348 lr: 0.001679 max_mem: 5632M
[03/18 08:32:13 d2.utils.events]: eta: 6:26:15 iter: 1700 total_loss: 5.497 loss_box2d_reg: 0.3394 loss_box3d_depth: 1.937 loss_box3d_proj_ctr: 0.2919 loss_box3d_quat: 1.165 loss_box3d_size: 0.1506 loss_centerness: 0.6444 loss_cls: 0.3255 loss_conf3d: 0.6378 lr: 0.001699 max_mem: 5632M
[03/18 08:32:33 d2.utils.events]: eta: 6:22:42 iter: 1720 total_loss: 5.457 loss_box2d_reg: 0.3388 loss_box3d_depth: 1.955 loss_box3d_proj_ctr: 0.2799 loss_box3d_quat: 1.086 loss_box3d_size: 0.1483 loss_centerness: 0.644 loss_cls: 0.3155 loss_conf3d: 0.6394 lr: 0.001719 max_mem: 5632M
[03/18 08:32:51 d2.utils.events]: eta: 5:59:36 iter: 1740 total_loss: 5.458 loss_box2d_reg: 0.3355 loss_box3d_depth: 1.962 loss_box3d_proj_ctr: 0.285 loss_box3d_quat: 1.095 loss_box3d_size: 0.1466 loss_centerness: 0.6434 loss_cls: 0.3263 loss_conf3d: 0.6384 lr: 0.001739 max_mem: 5632M
[03/18 08:33:10 d2.utils.events]: eta: 6:11:12 iter: 1760 total_loss: 5.344 loss_box2d_reg: 0.3334 loss_box3d_depth: 1.909 loss_box3d_proj_ctr: 0.2814 loss_box3d_quat: 1.049 loss_box3d_size: 0.1418 loss_centerness: 0.6441 loss_cls: 0.3185 loss_conf3d: 0.6423 lr: 0.001759 max_mem: 5632M
[03/18 08:33:29 d2.utils.events]: eta: 6:07:56 iter: 1780 total_loss: 5.221 loss_box2d_reg: 0.3335 loss_box3d_depth: 1.867 loss_box3d_proj_ctr: 0.2846 loss_box3d_quat: 1.031 loss_box3d_size: 0.1468 loss_centerness: 0.6424 loss_cls: 0.3144 loss_conf3d: 0.6463 lr: 0.001779 max_mem: 5632M
[03/18 08:33:47 d2.utils.events]: eta: 5:46:48 iter: 1800 total_loss: 5.386 loss_box2d_reg: 0.3327 loss_box3d_depth: 1.915 loss_box3d_proj_ctr: 0.2877 loss_box3d_quat: 1.108 loss_box3d_size: 0.1585 loss_centerness: 0.6427 loss_cls: 0.3144 loss_conf3d: 0.6437 lr: 0.001799 max_mem: 5632M
[03/18 08:34:06 d2.utils.events]: eta: 5:50:54 iter: 1820 total_loss: 5.412 loss_box2d_reg: 0.3317 loss_box3d_depth: 2.006 loss_box3d_proj_ctr: 0.2895 loss_box3d_quat: 1.022 loss_box3d_size: 0.1489 loss_centerness: 0.6423 loss_cls: 0.3128 loss_conf3d: 0.6448 lr: 0.001819 max_mem: 5632M
[03/18 08:34:24 d2.utils.events]: eta: 6:02:58 iter: 1840 total_loss: 5.868 loss_box2d_reg: 0.3389 loss_box3d_depth: 2.383 loss_box3d_proj_ctr: 0.2912 loss_box3d_quat: 1.055 loss_box3d_size: 0.1456 loss_centerness: 0.6443 loss_cls: 0.3225 loss_conf3d: 0.6344 lr: 0.001839 max_mem: 5632M
[03/18 08:34:42 d2.utils.events]: eta: 5:39:11 iter: 1860 total_loss: 5.477 loss_box2d_reg: 0.3404 loss_box3d_depth: 2.04 loss_box3d_proj_ctr: 0.2862 loss_box3d_quat: 1.025 loss_box3d_size: 0.148 loss_centerness: 0.6442 loss_cls: 0.3221 loss_conf3d: 0.6452 lr: 0.001859 max_mem: 5632M
[03/18 08:35:00 d2.utils.events]: eta: 5:49:06 iter: 1880 total_loss: 5.558 loss_box2d_reg: 0.3291 loss_box3d_depth: 2.17 loss_box3d_proj_ctr: 0.2895 loss_box3d_quat: 0.989 loss_box3d_size: 0.1437 loss_centerness: 0.6424 loss_cls: 0.3169 loss_conf3d: 0.638 lr: 0.001879 max_mem: 5632M
[03/18 08:35:19 d2.utils.events]: eta: 6:08:45 iter: 1900 total_loss: 5.253 loss_box2d_reg: 0.3289 loss_box3d_depth: 1.898 loss_box3d_proj_ctr: 0.2903 loss_box3d_quat: 0.9442 loss_box3d_size: 0.1506 loss_centerness: 0.6417 loss_cls: 0.3157 loss_conf3d: 0.6479 lr: 0.001899 max_mem: 5632M
[03/18 08:35:37 d2.utils.events]: eta: 5:41:41 iter: 1920 total_loss: 5.343 loss_box2d_reg: 0.3279 loss_box3d_depth: 1.974 loss_box3d_proj_ctr: 0.2896 loss_box3d_quat: 0.9662 loss_box3d_size: 0.1469 loss_centerness: 0.6409 loss_cls: 0.313 loss_conf3d: 0.6459 lr: 0.001919 max_mem: 5632M
[03/18 08:35:56 d2.utils.events]: eta: 6:06:53 iter: 1940 total_loss: 5.245 loss_box2d_reg: 0.3287 loss_box3d_depth: 1.981 loss_box3d_proj_ctr: 0.2698 loss_box3d_quat: 0.9367 loss_box3d_size: 0.1429 loss_centerness: 0.6427 loss_cls: 0.3073 loss_conf3d: 0.6487 lr: 0.001939 max_mem: 5632M
[03/18 08:36:16 d2.utils.events]: eta: 6:22:55 iter: 1960 total_loss: 5.244 loss_box2d_reg: 0.3229 loss_box3d_depth: 1.9 loss_box3d_proj_ctr: 0.2822 loss_box3d_quat: 1.009 loss_box3d_size: 0.1435 loss_centerness: 0.6422 loss_cls: 0.3102 loss_conf3d: 0.6472 lr: 0.001959 max_mem: 5632M
[03/18 08:36:34 d2.utils.events]: eta: 5:43:01 iter: 1980 total_loss: 5.142 loss_box2d_reg: 0.3235 loss_box3d_depth: 1.81 loss_box3d_proj_ctr: 0.2837 loss_box3d_quat: 0.945 loss_box3d_size: 0.1464 loss_centerness: 0.6411 loss_cls: 0.3118 loss_conf3d: 0.6522 lr: 0.001979 max_mem: 5632M
[03/18 08:36:52 d2.utils.events]: eta: 5:50:43 iter: 2000 total_loss: 5.204 loss_box2d_reg: 0.3229 loss_box3d_depth: 1.922 loss_box3d_proj_ctr: 0.2814 loss_box3d_quat: 0.9451 loss_box3d_size: 0.1452 loss_centerness: 0.6421 loss_cls: 0.3056 loss_conf3d: 0.6506 lr: 0.001999 max_mem: 5632M
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # 8000 type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
[03/18 08:37:00 fvcore.common.checkpoint]: Saving checkpoint to ./model_0001999.pth
[03/18 08:37:01 tridet.data.dataset_mappers.dataset_mapper]: Augmentations used in training: [ResizeShortestEdge(short_edge_length=(384, 384), max_size=100000, sample_style='choice')]
[03/18 08:37:01 d2.data.common]: Serializing 3769 elements to byte tensors and concatenating them all ...
[03/18 08:37:01 d2.data.common]: Serialized dataset takes 13.38 MiB
[03/18 08:37:01 tridet.data.build]: Using test sampler InferenceSampler
[03/18 08:37:01 d2.evaluation.evaluator]: Start inference on 48 batches
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()beforeoptimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()beforelr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:87: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
[03/18 08:37:21 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3040 s/iter. Eval: 1.2617 s/iter. Total: 1.5665 s/iter. ETA=0:00:57
[03/18 08:37:27 d2.evaluation.evaluator]: Inference done 15/48. Dataloading: 0.0009 s/iter. Inference: 0.3116 s/iter. Eval: 1.2645 s/iter. Total: 1.5771 s/iter. ETA=0:00:52
[03/18 08:37:32 d2.evaluation.evaluator]: Inference done 18/48. Dataloading: 0.0010 s/iter. Inference: 0.3153 s/iter. Eval: 1.3108 s/iter. Total: 1.6273 s/iter. ETA=0:00:48
[03/18 08:37:39 d2.evaluation.evaluator]: Inference done 22/48. Dataloading: 0.0010 s/iter. Inference: 0.3119 s/iter. Eval: 1.3120 s/iter. Total: 1.6249 s/iter. ETA=0:00:42
[03/18 08:37:45 d2.evaluation.evaluator]: Inference done 26/48. Dataloading: 0.0010 s/iter. Inference: 0.3091 s/iter. Eval: 1.3096 s/iter. Total: 1.6198 s/iter. ETA=0:00:35
[03/18 08:37:52 d2.evaluation.evaluator]: Inference done 30/48. Dataloading: 0.0010 s/iter. Inference: 0.3094 s/iter. Eval: 1.3118 s/iter. Total: 1.6223 s/iter. ETA=0:00:29
[03/18 08:37:58 d2.evaluation.evaluator]: Inference done 34/48. Dataloading: 0.0010 s/iter. Inference: 0.3091 s/iter. Eval: 1.3145 s/iter. Total: 1.6248 s/iter. ETA=0:00:22
[03/18 08:38:04 d2.evaluation.evaluator]: Inference done 38/48. Dataloading: 0.0010 s/iter. Inference: 0.3088 s/iter. Eval: 1.3051 s/iter. Total: 1.6150 s/iter. ETA=0:00:16
[03/18 08:38:10 d2.evaluation.evaluator]: Inference done 42/48. Dataloading: 0.0010 s/iter. Inference: 0.3082 s/iter. Eval: 1.2912 s/iter. Total: 1.6005 s/iter. ETA=0:00:09
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
[03/18 08:38:16 d2.evaluation.evaluator]: Inference done 46/48. Dataloading: 0.0010 s/iter. Inference: 0.3067 s/iter. Eval: 1.2753 s/iter. Total: 1.5832 s/iter. ETA=0:00:03
[03/18 08:38:18 d2.evaluation.evaluator]: Total inference time: 0:01:06.936973 (1.556674 s / iter per device, on 8 devices)
[03/18 08:38:18 d2.evaluation.evaluator]: Total inference pure compute time: 0:00:13 (0.304431 s / iter per device, on 8 devices)
################coco_evaluation.py evaluate
[03/18 08:38:22 d2.evaluation.coco_evaluation]: @@@@@@@@@@@@@@@@@###############Preparing results for COCO format ...
[03/18 08:38:22 d2.evaluation.coco_evaluation]: Saving results to /home/azuryl/dd3d_test/omni/dd3d/outputs/2023-03-18/08-05-55/inference/step0002000/kitti_3d_val/coco_instances_results.json
[03/18 08:38:24 d2.evaluation.coco_evaluation]: Evaluating predictions with unofficial COCO API...
Loading and preparing results...
DONE (t=1.02s)
creating index...
index created!
[03/18 08:38:25 d2.evaluation.fast_eval_api]: Evaluate annotation type bbox
[03/18 08:38:29 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 3.81 seconds.
[03/18 08:38:29 d2.evaluation.fast_eval_api]: Accumulating evaluation results...
[03/18 08:38:30 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 1.12 seconds.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.062
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.148
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.043
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.067
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.065
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.081
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.083
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.206
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.254
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.135
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.273
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.300
[03/18 08:38:30 d2.evaluation.coco_evaluation]: Evaluation results for bbox:
100%|██████████| 20/20 [00:07<00:00, 2.74it/s][03/18 08:40:01 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 08:40:01 tridet.utils.train]: Test results:
The text was updated successfully, but these errors were encountered: