8000 net/gcoap: port to ztimer by kfessel · Pull Request #17141 · RIOT-OS/RIOT · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

net/gcoap: port to ztimer #17141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 9, 2021
Merged

net/gcoap: port to ztimer #17141

merged 6 commits into from
Dec 9, 2021

Conversation

kfessel
Copy link
Contributor
@kfessel kfessel commented Nov 5, 2021

Contribution description

switches gcoap to use ztimer

Testing procedure

run and manualy test with examples/gcoap

Issues/PRs references

waiting for #16958 which would enable ztimer_msec use

@github-actions github-actions bot added Area: CoAP Area: Constrained Application Protocol implementations Area: network Area: Networking Area: sys Area: System labels Nov 5, 2021
@kfessel kfessel added the Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation label Nov 5, 2021
@chrysn
Copy link
Member
chrysn commented Nov 5, 2021

That's great, and could profit from the plans over on #16413 to avoid race conditions that are currently there somewhere.

@kfessel
Copy link
Contributor Author
kfessel commented Nov 17, 2021

@Crysn: There is one USEC left for GCOAP_OBS_TICK can we avoid these?

@chrysn
Copy link
Member
chrysn commented Nov 17, 2021

Yes, but not in this PR. Observation using time based observe values is not really practical, should be incrementing on its own scale on demand. Changing this is probably best done after this PR because a) it is then well motivated, and b) not creating cross-PR conflicts.

@kfessel kfessel marked this pull request as ready for review November 17, 2021 16:23
@fjmolinas fjmolinas added this to the Release 2022.01 milestone Nov 18, 2021
Comment on lines +1270 to 1271
uint32_t now = ztimer_now(ZTIMER_USEC);
pdu->observe_value = (now >> GCOAP_OBS_TICK_EXPONENT) & 0xFFFFFF;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we could get rid of the ZTIMER_USEC dependency here, if RTT is used and running at 32Khz, then the base timer tick would be 30us. This brings the need for this feature to be exposed again though... So I guess that would be a follow-up, but in that case ztimer_usec could be avoided altogether for gcoap..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definetly another pr (maybe we even do not need that see chrysn 2021-11-17)

also: be careful with increasing frequencys they may lead to needing extra module (ztimer_extended) additional wakeups (for that extended)
i am also not sure if we consider ztimers bases open for non ztimer use

Digest: lets get it to ztimer and later improve its ztimer use.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definetly another pr (maybe we even do not need that see chrysn 2021-11-17)

I think you forgot the link here :)

8000

Copy link
Contributor Author
@kfessel kfessel Nov 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kfessel
Copy link
Contributor Author
kfessel commented Nov 29, 2021

@fjmolinas @chrysn: Does one of you want to do the review?

Copy link
Contributor
@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

8000

The reason will be displayed to describe this comment to others. Learn more.

A couple of nitpicks otherwise this looks good. Could you run some tests o this though?

Comment on lines -1012 to +1014
uint32_t start = xtimer_now_usec();
res = xtimer_msg_receive_timeout(&msg, timeout);
uint32_t start = ztimer_now(ZTIMER_MSEC);
res = ztimer_msg_receive_timeout(ZTIMER_MSEC, &msg, timeout);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The units where changed here, are you sure msec are enough?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in all cases where i swap ed timeout of unamend unit (which would be usec in all cases) or usec i checked the typical default values if they mad to msec and only swaped if they did.
There is one exception GCOAP_OBS_TICK_EXPONENT that can not map to integer msec -> it is still usec

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also changed all default timeout to (_MSEC) if applicable (see gcoap.h) and ether devided them by 1000 or changed their time factor. (most of these time outs are multiple seconds and msec is used to have enough resolution
In cases this will also save us of the inclusion of ztimer_sec, which would lead to ztimer extended and that may use extra power.

Comment on lines +1270 to 1271
uint32_t now = ztimer_now(ZTIMER_USEC);
pdu->observe_value = (now >> GCOAP_OBS_TICK_EXPONENT) & 0xFFFFFF;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definetly another pr (maybe we even do not need that see chrysn 2021-11-17)

I think you forgot the link here :)

event_callback_init(&_dtls_session_free_up_tmout_cb,
_dtls_free_up_session, NULL);
event_timeout_init(&_dtls_session_free_up_tmout, &_queue,
event_timeout_ztimer_init(&_dtls_session_free_up_tmout, ZTIMER_MSEC,&_queue,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
event_timeout_ztimer_init(&_dtls_session_free_up_tmout, ZTIMER_MSEC,&_queue,
event_timeout_ztimer_init(&_dtls_session_free_up_tmout, ZTIMER_MSEC, &_queue,

@kfessel kfessel added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Nov 30, 2021
@kaspar030 kaspar030 added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 30, 2021
@kfessel kfessel added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Dec 1, 2021
@github-actions github-actions bot added Area: examples Area: Example Applications Area: tests Area: tests and testing framework labels Dec 2, 2021
@kfessel
Copy link
Contributor Author
kfessel commented Dec 3, 2021

@fjmolinas: i added the new ztimer_no_periph_rtt to the rtt_rtc unit test - it is silencing the warning ✔️
but i think even with the warning was generated the ztimer_periph_rtt should already have not be used due to the conflict with rtt_rtc

make: Entering directory '/tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/tests/unittests'
WARNING! The following modules conflict with 'ztimer_periph_rtt': 'rtt_rtc'
To disable ztimer periph_rtt auto-inclusion add 'ztimer_no_periph_rtt' to 'USEMODULE'
Building application "tests_unittests" for "acd52832" with MCU "nrf52".
rm -rf /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/pkg-build/tiny-asn1

[INFO] cloning tiny-asn1
git-cache: cloning from cache. tag=commit7005fcff4706e96b857f257ef94b8518211c9fbc-2004067
[INFO] updating tiny-asn1 /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/pkg/tiny-asn1/.pkg-state.git-downloaded
echo 7005fcff4706e96b857f257ef94b8518211c9fbc > /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/pkg/tiny-asn1/.pkg-state.git-downloaded
[INFO] patch tiny-asn1
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_get_counter':
mock_rtt.c:(.text.rtt_get_counter+0x0): multiple definition of `rtt_get_counter'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_get_counter+0x0): first defined here
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_set_alarm':
mock_rtt.c:(.text.rtt_set_alarm+0x0): multiple definition of `rtt_set_alarm'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_set_alarm+0x0): first defined here
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_get_alarm':
mock_rtt.c:(.text.rtt_get_alarm+0x0): multiple definition of `rtt_get_alarm'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_get_alarm+0x0): first defined here
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_clear_alarm':
mock_rtt.c:(.text.rtt_clear_alarm+0x0): multiple definition of `rtt_clear_alarm'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_clear_alarm+0x0): first defined here
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_set_overflow_cb':
mock_rtt.c:(.text.rtt_set_overflow_cb+0x0): multiple definition of `rtt_set_overflow_cb'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_set_overflow_cb+0x0): first defined here
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_clear_overflow_cb':
mock_rtt.c:(.text.rtt_clear_overflow_cb+0x0): multiple definition of `rtt_clear_overflow_cb'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_clear_overflow_cb+0x0): first defined here
/opt/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests-rtt_rtc/mock_rtt.o: in function `rtt_init':
mock_rtt.c:(.text.rtt_init+0x0): multiple definition of `rtt_init'; /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/nrf5x_common_periph/rtt.o:rtt.c:(.text.rtt_init+0x0): first defined here
collect2: error: ld returned 1 exit status
make: *** [/tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/tests/unittests/../../Makefile.include:686: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/tests_unittests.elf] Error 1
make: Leaving directory '/tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/tests/unittests'
cat: /tmp/dwq.0.2712567277966057/00cc4821a03ef7a5f80c40157c36a11e/build/test-input-hash.sha1: No such file or directory
-- build directory size: 6.4M
---
--- build output of app tests/unittests for board alientek-pandora:gnu (raw, runtime=5.0s):
-- running on worker mobi7-0 thread 3, build number 55141.

out-failed-only.md

@fjmolinas
Copy link
Contributor

but i think even with the warning was generated the ztimer_periph_rtt should already have not be used due to the conflict with rtt_rtc

I can't reproduce, maybe rebase this branch?

@kfessel
Copy link
Contributor Author
kfessel commented Dec 3, 2021

did you test with or without the last commit (which "fixes"/works around the issue)?

to reproduce merge or rebase PR with master, without the last commit "test/unittest: no auto ztimer_periph_rtt"
or just remove that line from: RIOT/tests/unittests/tests-rtt_rtc/Makefile.include

it seem like there is a problem in the conflict resolution

@fjmolinas
Copy link
Contributor

but i think even with the warning was generated the ztimer_periph_rtt should already have not be used due to the conflict with rtt_rtc

did you test with or without the last commit (which "fixes"/works around the issue)

Ahhh no I did not, but that is expected, the filtering is not done against conflicting modules, you must explicitly disable it. Filtering against it is another thing. Explicitly disabling it avoids dealing with the question of which one we should prefer. Could be improved upon later...

@kfessel
Copy link
Contributor Author
kfessel commented Dec 3, 2021

ahh ok i thought there is a automatic conflict management in the module system

@fjmolinas
Copy link
Contributor

ahh ok i thought there is a automatic conflict management in the module system

Nope we didn't add that , we just added the warning. There is conflict management for FEATURES, modules happen with ifeq filters but those do not work very well as dependency order makes them fail somtimes.

@kfessel
Copy link
Contributor Author
kfessel commented Dec 6, 2021

I just ran the gcoap example (native with gnrc-tap) after all these additional commits ✔️ -> ready for review
term 1:

$ coap-server

term 2:

RIOT$ dist/tools/tapsetup/tapsetup

/examples/gcoap$ make flash term
... building
main(): This is RIOT! (Version: 2021.10-devel-1729-gd7d1f-p-ztimer-gcoap)
gcoap example app
All up, running the shell now
> ifconfig
ifconfig
...
          inet6 addr: fe80::10b2:7eff:fe3b:a191  scope: link  VAL
          inet6 group: ff02::1
          inet6 group: ff02::1:ff3b:a191

> coap get fe80::10b2:7eff:fe3b:a190 5683 /.well-known/core
coap get fe80::10b2:7eff:fe3b:a190 5683 /.well-known/core
gcoap_cli: sending msg ID 61392, 23 bytes
> gcoap: response Success, code 2.05, 105 bytes
</>;title="General Info";ct=0,</time>;if="clock";rt="ticks";title="Internal Clock";ct=0;obs,</async>;ct=0

term 3:

$ coap-client -N -m get -p 5683 coap://[ff02::1:ff3b:a191%tap0]/cli/stats
2

@kfessel kfessel requested a review from chrysn December 6, 2021 15:36
@@ -26,6 +26,7 @@
#include "net/cord/common.h"
#include "net/sock/util.h"
#include "net/ipv6/addr.h"
#include "xtimer.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add xtimer here if the PR is about removing xtimer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this example is using xtimer but didn't include it (was used via gcoap.h)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, won't block it in any case, I would have just have gone the other way around

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, won't block it in any case, I would have just have gone the other way around

@fjmolinas
Copy link
Contributor

I just ran the gcoap example (native with gnrc-tap) after all these additional commits heavy_check_mark -> ready for review

It would be awesome if you provided some test output when tests are ran, its good to have some history on the PR

@fjmolinas
Copy link
Contributor

It would be awesome if you provided some test output when tests are ran, its good to have some history on the PR

Timeouts and DTLS would need testing I guess

@kfessel
Copy link
Contributor Author
kfessel commented Dec 7, 2021

i updated that comment does that help?

I never used DTLS

$ coap-client -N -m get -p 5684 coaps://[ff02::1:ff3b:a191%tap0]/cli/stats
Dec 07 15:52:55 EMRG coaps URI scheme not supported in this version of libcoap

seems my libcoap doesn't support it either

@fjmolinas
Copy link
Contributor

Maybe @leandrolanzieri could test this one?

Copy link
Contributor
@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing timeout is still 5s.

2021-12-08 20:24:51,172 # coap post fe80::ac8d:fee1:e02c:e32f:5 5683 /tmp "hello"
2021-12-08 20:24:51,174 # gcoap_cli: sending msg ID 43790, 17 bytes
> 2021-12-08 20:24:56,180 # gcoap: timeout for msg ID 43790

I tried testing dtls but never got a response or a timeout, @leandrolanzieri should this work out of the box when communicating between two nodes?

@leandrolanzieri
Copy link
Contributor

Testing timeout is still 5s.

2021-12-08 20:24:51,172 # coap post fe80::ac8d:fee1:e02c:e32f:5 5683 /tmp "hello"
2021-12-08 20:24:51,174 # gcoap_cli: sending msg ID 43790, 17 bytes
> 2021-12-08 20:24:56,180 # gcoap: timeout for msg ID 43790

I tried testing dtls but never got a response or a timeout, @leandrolanzieri should this work out of the box when communicating between two nodes?

Yes, did you use port 5684?

I was not particularly involved during the integration of DTLS to gcoap but it seems that indeed when the handshake fails there are no retries nor message is shown. We should look into that, but it is unrelated to the PR as it occurs on master as well.

I tested with this PR and the example still works as expected, but I can't say anything about timeouts:

main(): This is RIOT! (Version: 2022.01-devel-915-gd7d1f-p-ztimer-gcoap)
gcoap example app
All up, running the shell now
> coap get fe80::7081:f6ff:fe7c:f496%7 5684 /.well-known/core
coap get fe80::7081:f6ff:fe7c:f496%7 5684 /.well-known/core
gcoap_cli: sending msg ID 47824, 23 bytes
> gcoap: response Success, code 2.05, 46 bytes
</cli/stats>;ct=0;rt="count";obs,</riot/board>

@fjmolinas
Copy link
Contributor

Yes, did you use port 5684?

Yes it seems it does not work for me with real hardware, but it does with native..

I was not particularly involved during the integration of DTLS to gcoap but it seems that indeed when the handshake fails there are no retries nor message is shown. We should look into that, but it is unrelated to the PR as it occurs on master as well.

No message seems to be a matter of enabling debug messages though..., after enabling I verified the handshake timeout is OK:

main(): This is RIOT! (Version: 2022.01-devel-915-gb200f-pr-17141)
gcoap example app
All up, running the shell now
> coap get fe80::4c4c:91ff:fe2e:14db 5684 /.well-known/core
coap get fe80::4c4c:91ff:fe2e:14db 5684 /.well-known/core
gcoap_cli: sending msg ID 16773, 23 bytes
gcoap: authentication timed out
gcoap: sock send failed: -107
>  coap get fe80::4c4c:91ff:fe2e:14db 5684 /.well-known/core
 coap get fe80::4c4c:91ff:fe2e:14db 5684 /.well-known/core
gcoap_cli: sending msg ID 16774, 23 bytes
gcoap: authentication timed out
gcoap: sock send failed: -107

Copy link
Contributor
@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observing still works:

→ aiocoap-client -m GET "coap://[2001:db8::2]/cli/stats" --observe
0
---
1
---
2
main(): This is RIOT! (Version: 2022.01-devel-915-gb200f-pr-17141)
gcoap example app
All up, running the shell now
> ifconfig 6 add 2001:db8::2/64
ifconfig 6 add 2001:db8::2/64
success: added 2001:db8::2/64 to interface 6
> gcoap: Registered observer for: /cli/stats
coap get fe80::4c4c:91ff:fe2e:14da 5684 /cli/stats
coap get fe80::4c4c:91ff:fe2e:14da 5684 /cli/stats
gcoap_cli: sending msg ID 539, 16 bytes
gcoap_cli: creating /cli/stats notification
> coap: received timeout message
gcoap: timeout for msg ID 539
coap get fe80::4c4c:91ff:fe2e:14da 5684 /cli/stats
coap get fe80::4c4c:91ff:fe2e:14da 5684 /cli/stats
gcoap_cli: sending msg ID 541, 16 bytes
gcoap_cli: creating /cli/stats notification
> coap: received timeout message
gcoap: timeout for msg ID 541

Copy link
Contributor
@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK!, I've tested what seemed testable. There is maybe an issue with dtls on real hardware but this is also in master... I'll open an issue once I get more info from the initial PR contributor.

Otherwise this makes me think that it's a shame https://github.com/kb2ma/riot-coap-pytest where never integrated in tree or in the release, maybe @miri64 might be interested in those since she has been working witj gcoap a bunch lately.

@kfessel kfessel merged commit 6347dcd into RIOT-OS:master Dec 9, 2021
@kfessel
Copy link
Contributor Author
kfessel commented Dec 9, 2021

thank you for reviewing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: CoAP Area: Constrained Application Protocol implementations Area: examples Area: Example Applications Area: network Area: Networking Area: sys Area: System Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants
0