Refactor detr 3.x conditional detr and group detr #9248
+3,936
−2,515
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* [Fix] Fix UT to be compatible with pytorch 1.6 (open-mmlab#8707) * Update * Update * force reinstall pycocotools * Fix build_cuda * docker install git * Update * comment other job to speedup process * update * uncomment * Update * Update * Add comments for --force-reinstall * [Refactor] Refactor anchor head and base head with boxlist (open-mmlab#8625) * Refactor anchor head * Update * Update * Update * Add a series of boxes tools * Fix box type to support n x box_dim boxes * revert box type changes * Add docstring * refactor retina_head * Update * Update * Fix comments * modify docstring of coder and ioucalculator * Replace with_boxlist with use_box_type * fix: fix config of detr-r18 * fix: modified import of MSDeformAttn in PixelDecoder of Mask2Former * feat: add TransformerDetector as the base detector of DETR-like detectors * refactor: refactor modules and configs of DETR * refactor: refactor DETR-related modules in transformer.py * refactor: refactor DETR-related modules in transformer.py * fix: add type comments in detr.py * correct trainloop in detr_r50 config * fix: modify the parent class of DETRHead to BaseModule * refactor: refactor modules and configs of Deformable DETR * fix: modify the usage of num_query * fix: modify the usage of num_query in configs * refactor: replace input_proj of detr with ChannelMapper neck * refactor: delete multi_apply in DETRHead.forward() * Update detr_r18_8xb2-500e_coco.py using channel mapper for r18 * change the name of detection_transfomer.py to base_detr.py * refactor: modify construct binary masks section of forward_pretransformer * refactor: utilize abstractmethod * update ABCmeta to make sure reload class TransformerDetector * some annotation * some annotation * some annotation * refactor: delete _init_transformer in detectors * refactor: modify args of deformable detr * refactor: modify about super().__init__() * Update detr_head.py Remove the multi feat lvl in function 'predict_by_feat' * Update detr.py update init_weights * some annotation for head * to make sure the head args the same as detector * to make sure the head args the same as detector * some bug * fix: fix bugs of num_pred in DeformableDETRHead * add kwargs to transformer * support MLP and sineembed position * detele positional encodeing * delete useless postnorm * Revert "add kwargs to transformer" This reverts commit a265c1a. * Update detr_head.py Update type and shape of args * Update detr_head.py fix args docstring in predict_by_feat * Update base_detr.py Update docstring for forward_pretransformer * Update deformable_detr.py Fix docstring * to support conditional detr with reload forward_transformer * fix: update config files of Two-stage and Box-refine * replace all bs with batch_size in detr-related files * update deformable.py and transformer.py * update docstring in base_detr * update docstring in base_detr, detr * doc refine * Revert "doc refine" This reverts commit b69da4f. * doc refine * doc refine * updabase_detr, detr, and le layers/transformdoc * fix doc in base_detr * add origin repo link * add origin repo link * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * doc: add doc of the first edition of Deformable DETR * batch_size to bs * refine doc * refine doc * feat: add config comments of specific module * refactor: refactor base DETR class TransformerDetector * fix: fix wrong return typehint of forward_encoder in TransformerDetector * refactor: refactor DETR * refactor: refactor Deformable DETR * refactor: refactor forward_encoder and pre_decoder * fix: fix bugs of new edition * refactor: small modifications * fix: move get_reference_points to deformable_encoder * refactor: merge init_&inter_reference to references in Deformable DETR * modify docstring of get_valid_ratio in Deformable DETR * add some docstring * doc: add docstring of deformable_detr.py * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of base_detr.py * doc: refine docstring of base_detr.py * doc: refine docstring of base_detr.py * a little change of MLP * a little change of MLP * a little change of MLP * a little change of MLP * refine config * refine config * refine config * refine doc string for detr * little refine doc string for detr.py * tiny modification * doc: refine docstring of detr.py * tiny modifications to resolve the conversations * DETRHead.predict() draft * tiny modifications to resolve conversations * refactor: modify arg names and forward strategies of bbox_head * tiny modifications to resolve the conversations * support MLP * fix docsting of function pre_decoder * fix docsting of function pre_decoder * fix docstring * modifications for resolving conversations * refactor: eradicate key_padding_mask args * refactor: eradicate key_padding_mask args * fix: fix bug of deformable detr and resolve some conversations * refactor: rename base class with DetectionTransformer and other modifications * fix: fix config of detr * fix the bug of init * fix: fix init_weight of DETR and Deformable DETR * resolve conflict * fix auto-merge bug * fix pre-commit bug * refactor: move the position of encoder and decoder * delete Transformer in ci test * delete Transformer in ci test Co-authored-by: jbwang1997 <jbwang1997@gmail.com> Co-authored-by: KeiChiTse <xqz20@mails.tsinghua.edu.cn> Co-authored-by: LYMDLUT <70597027+LYMDLUT@users.noreply.github.com> Co-authored-by: lym <letusgo126@126.com> Co-authored-by: Kei-Chi Tse <109070650+KeiChiTse@users.noreply.github.com>
…tr' into refactor-detr-3.x-rebase-groupdetr8000
Li-Qingyun
reviewed
Nov 6, 2022
Comment on lines
1
to
35
| _base_ = [ | ||
| '../detr/detr_r50_8xb2-150e_coco.py' | ||
| ] | ||
| model = dict( | ||
| type='ConditionalDETR', | ||
| num_query=300, | ||
| decoder=dict( | ||
| num_layers=6, | ||
| layer_cfg=dict( | ||
| self_attn_cfg=dict(embed_dims=256, num_heads=8, dropout=0.1, cross_attn=False), | ||
| cross_attn_cfg=dict(embed_dims=256, num_heads=8, dropout=0.1, cross_attn=True))), | ||
| bbox_head=dict( | ||
| type='ConditionalDETRHead', | ||
| loss_cls=dict( | ||
| _delete_=True, | ||
| type='FocalLoss', | ||
| use_sigmoid=True, | ||
| gamma=2.0, | ||
| alpha=0.25, | ||
| loss_weight=2.0)), | ||
| # training and testing settings | ||
| train_cfg=dict( | ||
| assigner=dict( | ||
| type='HungarianAssigner', | ||
| match_costs=[ | ||
| dict(type='FocalLossCost', weight=2.0), | ||
| dict(type='BBoxL1Cost', weight=5.0, box_format='xywh'), | ||
| dict(type='IoUCost', iou_mode='giou', weight=2.0) | ||
| ]))) | ||
|
|
||
| # learning policy | ||
| max_epochs = 50, | ||
|
|
||
| param_scheduler = [dict(type='MultiStepLR', milestones=[40])] | ||
|
|
There was a problem hiding this comment.
Does the config of DINO require to inherit the Deformable DETR config? @jshilong
There was a problem hiding this comment.
I prefer the inheritance approach, but is it supposed to have at least one config in an algorithm without too much inheritance?
Li-Qingyun
reviewed
Nov 6, 2022
3 tasks
jshilong
reviewed
Nov 27, 2022
| """ | ||
|
|
||
| def __init__(self, *arg, group_detr=1, **kwargs) -> None: | ||
| self.group_detr = group_detr |
There was a problem hiding this comment.
move all group detr related code to another pr
jshilong
reviewed
Nov 27, 2022
| for layer_id in range(self.num_layers - 1): | ||
| self.layers[layer_id + 1].cross_attn.qpos_proj = None | ||
|
|
||
| def forward(self, query: Tensor, key: Tensor, value: Tensor, |
There was a problem hiding this comment.
add doc
jshilong
reviewed
Nov 27, 2022
|
|
||
| nn.init.constant_(self.out_proj.bias, 0.) # init out_proj | ||
|
|
||
| def forward_attn(self, |
jshilong
reviewed
Nov 27, 2022
| self._init_proj() | ||
| self.group_detr = group_detr | ||
|
|
||
| def _init_proj(self): |
07d7ae4 to
4c11117
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement the results in the paper "[Conditional DETR: Conditional DETR for Fast Training Convergence]" and "[Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment]"with mmdetection. The original code has released.
Remark
Besides, there are also brother PRs #9149, #8544, #8846, #8533, #8811 of three algorithms (DINO、old Condition DETR and DAB DETR) developed in parallel.
This is my first experience of contributing a PR to the open source community. I'm trying hard to make Conditional DETR and Group DETR available in mmdetection as soon as possible.
The mAP of Group DETR is still aligning，do not worry，it will release in very recent days.