[BugFixed] fix wrong trunc_normal_init use #6432

vealocia · 2021-11-03T03:31:44Z

AronLin · 2021-11-03T10:58:49Z

mmdet/models/backbones/pvt.py

codecov · 2021-11-04T04:19:09Z

RangiLyu

hhaAndroid · 2021-11-04T06:28:57Z

mmdet/models/backbones/pvt.py

@@ -275,7 +276,7 @@ def __init__(self, pos_shape, pos_dim, drop_rate=0., init_cfg=None):
        self.drop = nn.Dropout(p=drop_rate)

    def init_weights(self):
-        trunc_normal_init(self.pos_embed, std=0.02)
+        trunc_normal_(self.pos_embed, std=0.02)


What is the difference between the two?

trunc_normal_init requires input module must be nn.Module, neither torch.Tensor nor nn.Parameter. If the input is torch.Tensor, the tensor will not be init as it should be.

for more details, please refer to trunc_normal_init

What is the difference between the two?

trunc_normal_init can only init module with weight and bias, not nn.Parameter. The position embedding is not initialized correctly now.

O, I get it. It is best to add a note to explain the precautions, because it is very easy to confuse.

Maybe it's better to add some warning in MMCV, two implementations of trunc init are really really confused.

Maybe it's better to add some warning in MMCV, two implementations of trunc init are really really confused.

Agree. Welcome to create a PR to MMCV.

Junjun2016 · 2021-11-24T12:22:40Z

mmdet/models/backbones/swin.py

                    constant_init(m.bias, 0)
                    constant_init(m.weight, 1.0)


constant_init is also for modules

Junjun2016 · 2021-12-02T03:58:58Z

mmdet/models/backbones/pvt.py

-                    trunc_normal_init(m.weight, std=.02)
-                    if m.bias is not None:
-                        constant_init(m.bias, 0)
+                    trunc_normal_init(m, std=.02, bias=0.)
                elif isinstance(m, nn.LayerNorm):
                    constant_init(m.bias, 0)


constant_init is also used for nn.Module.

Junjun2016 · 2021-12-02T04:02:31Z

mmdet/models/backbones/pvt.py

-                    trunc_normal_init(m.weight, std=.02)
-                    if m.bias is not None:
-                        constant_init(m.bias, 0)
+                    trunc_normal_init(m, std=.02, bias=0.)


normal_init is also used for nn.Module.

fix wrong trunc_normal_init use

adffe56

ZwwWayne requested a review from RangiLyu November 3, 2021 04:04

RangiLyu reviewed Nov 4, 2021

View reviewed changes

mmdet/models/backbones/pvt.py Outdated Show resolved Hide resolved

RangiLyu added the bug Something isn't working label Nov 4, 2021

fix wrong trunc_normal_init use

d4f5f36

RangiLyu self-requested a review November 4, 2021 04:25

RangiLyu approved these changes Nov 4, 2021

View reviewed changes

RangiLyu changed the base branch from master to dev-v2.18.1 November 4, 2021 05:26

hhaAndroid reviewed Nov 4, 2021

View reviewed changes

hhaAndroid approved these changes Nov 4, 2021

View reviewed changes

ZwwWayne changed the base branch from dev-v2.18.1 to master November 8, 2021 12:32

ZwwWayne merged commit 6cf9aa1 into open-mmlab:master Nov 8, 2021

RangiLyu mentioned this pull request Nov 9, 2021

Iteration Plan of v2.19.0 - November 2021 #6421

Closed

24 tasks

MengzhangLI mentioned this pull request Nov 24, 2021

[Fix] fix wrong trunc_normal_init use open-mmlab/mmsegmentation#1067

Closed

Junjun2016 reviewed Nov 24, 2021

View reviewed changes

MengzhangLI mentioned this pull request Nov 25, 2021

[Fix] Fix wrong init usage in transformer models open-mmlab/mmsegmentation#1069

Merged

F438
Junjun2016 reviewed Dec 2, 2021

View reviewed changes

Junjun2016 reviewed Dec 2, 2021

View reviewed changes

ZwwWayne pushed a commit that referenced this pull request Jul 18, 2022

[BugFixed] fix wrong trunc_normal_init use (#6432)

45d728c

* fix wrong trunc_normal_init use * fix wrong trunc_normal_init use

ZwwWayne pushed a commit to ZwwWayne/mmdetection that referenced this pull request Jul 19, 2022

[BugFixed] fix wrong trunc_normal_init use (open-mmlab#6432)

0e9d535

* fix wrong trunc_normal_init use * fix wrong trunc_normal_init use

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BugFixed] fix wrong trunc_normal_init use #6432

[BugFixed] fix wrong trunc_normal_init use #6432

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[BugFixed] fix wrong trunc_normal_init use #6432

[BugFixed] fix wrong trunc_normal_init use #6432

Uh oh!

Conversation

Motivation

Modification

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!