Avoid protobuf message construction when tx buffer is full #8787

bdraco · 2025-05-14T01:38:49Z

What does this implement/fix?

This PR refactors the api_connection to avoid protobuf message construction when the transmit buffer is full.

The code in api_connection.cpp has been restructured to change how messages are sent, avoiding the need to add buffer-full checks in every single send function. This approach is cleaner and solves the problem at its source.

Avoids constructing protobuf messages that would fail to send due to a full tx_buffer
Eliminates the wasteful cycle of:
1. Constructing messages
2. Failing to send
3. Queuing
4. Destroying the failed message
5. Reconstructing the message when retrying
Functional changes are
1. try_to_clear_buffer calls in the new helpers which try again to send the buffer if can_write_without_blocking if false before proceeding to queue the message.
2. Wrap process_queue with a can_write_without_blocking check
3. All the try_ functions are now marked as protected because it didn't seem like the intent was to make them part of the public API

Testing

external_components:
  - source: github://pr#8787
    components: [ api ]
    refresh: 0s

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Code quality improvements to existing code or addition of tests
Other

Related issue or feature (if applicable):

fixes

Pull request in esphome-docs with documentation (if applicable):

esphome/esphome-docs#

Test Environment

Example entry for `config.yaml`:

# Example config.yaml

Checklist:

The code change is tested and works locally.
Tests have been added to verify that the new code works (under tests/ folder).

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated in esphome-docs.

…e tx_buffer is full Also do not try to dequeue if the buffer is full

esphome/components/api/api_connection.h

probot-esphome · 2025-05-14T05:38:38Z

Hey there @OttoWinter, mind taking a look at this pull request as it has been labeled with an integration (api) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

esphome/components/api/api_connection.cpp

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

jesserockz

Nice logical changes and cleanups.
Testing now on some devices

jesserockz · 2025-05-15T00:16:21Z

Working on a handful of devices and one with a bunch of sensors that all send at the same time

bdraco · 2025-05-15T02:25:37Z

Thanks.

Some more dummy entities I used for testing

button:
  - platform: factory_reset
    id: factory_reset_btn
    name: Factory reset

sensor:
  # Uptime sensor.
  - platform: uptime
    name: Ethernet Uptime
  - platform: template
    name: Free Memory
    lambda: return heap_caps_get_free_size(MALLOC_CAP_INTERNAL);
    unit_of_measurement: 'B'
    state_class: measurement
  - platform: template
    id: test_climate_sensor
    lambda: return 20.0;

binary_sensor:
  - platform: template
    name: "Test Binary Sensor"
    lambda: return false;

cover:
  - platform: time_based
    name: "Test Cover"
    open_duration: 5s
    close_duration: 5s
    open_action:
      - logger.log: "Cover opening"
    close_action:
      - logger.log: "Cover closing"
    stop_action:
      - logger.log: "Cover stopping"

fan:
  - platform: binary
    output: test_light_output
    name: "Test Fan"

switch:
  - platform: template
    id: test_switch
    name: "Test Switch"
    lambda: return false;
    turn_on_action:
      - logger.log: "Switch turned on"
    turn_off_action:
      - logger.log: "Switch turned off"
      

text_sensor:
  - platform: version
    name: "ESPHome Version"

light:
  - platform: binary
    name: "Test Light"
    output: test_light_output

climate:
  - platform: bang_bang
    name: "Test Climate"
    sensor: test_climate_sensor
    default_target_temperature_low: 20°C
    default_target_temperature_high: 22°C
    heat_action:
      - switch.turn_on: test_switch
    idle_action:
      - switch.turn_off: test_switch

number:
  - platform: template
    name: "Test Number"
    min_value: 0
    max_value: 100
    step: 1
    lambda: return 50.0;
    set_action:
      - logger.log: "Number set"

select:
  - platform: template
    name: "Test Select"
    options:
      - "Option 1"
      - "Option 2"
    initial_option: "Option 1"
    optimistic: true
    set_action:
      - logger.log: "Select changed"

text:
  - platform: template
    name: "Test Text"
    mode: text
    initial_value: "Hello"
    set_action:
      - logger.log: "Text changed"

lock:
  - platform: output
    name: "Test Lock"
    output: test_light_output

output:
  - platform: gpio
    pin: GPIO0
    id: test_light_output
  - platform: gpio
    pin: GPIO2
    id: test_climate_heat
  - platform: gpio
    pin: GPIO3
    id: test_climate_cool

# Date, Time, and DateTime entities
datetime:
  - platform: template
    type: date
    name: "Test Date"
    initial_value: "2023-05-13"
    optimistic: true
    on_value:
      - logger.log: "Date changed"
  - platform: template
    type: time
    name: "Test Time"
    initial_value: "12:30:00"
    optimistic: true
    on_value:
      - logger.log: "Time changed"
  - platform: template
    type: datetime
    name: "Test DateTime"
    optimistic: true
    on_value:
      - logger.log: "DateTime changed"

# Valve
valve:
  - platform: template
    name: "Test Valve"
    open_action:
      - logger.log: "Valve opening"
    close_action:
      - logger.log: "Valve closing"
    stop_action:
      - logger.log: "Valve stopping"

# For now, commenting out media_player since it requires hardware support
# media_player:
#   - platform: speaker
#     name: "Test Media Player"

# Alarm Control Panel
alarm_control_panel:
  - platform: template
    name: "Test Alarm"
    codes:
      - "1234"
    arming_away_time: 0s
    arming_home_time: 0s
    pending_time: 0s
    trigger_time: 300s
    restore_mode: ALWAYS_DISARMED
    on_disarmed:
      - logger.log: "Alarm disarmed"
    on_arming:
      - logger.log: "Alarm arming"
    on_armed_away:
      - logger.log: "Alarm armed away"
    on_armed_home:
      - logger.log: "Alarm armed home"
    on_pending:
      - logger.log: "Alarm pending"
    on_triggered:
      - logger.log: "Alarm triggered"

# Event
event:
  - platform: template
    name: "Test Event"
    event_types:
      - first_event
      - second_event

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

) Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

Refactor api_connection to avoid constructing protobuf messages if th…

3fb1003

…e tx_buffer is full Also do not try to dequeue if the buffer is full

bdraco added 2 commits May 13, 2025 20:43

preen

85457ee

reduce diff

c0b9f64

bdraco changed the title ~~Refactor api_connection to trying to send messages if the tx_buffer is full~~ Refactor api_connection to avoid trying to send messages if the tx_buffer is full May 14, 2025

fixes

dcdc2a3

bdraco changed the title ~~Refactor api_connection to avoid trying to send messages if the tx_buffer is full~~ Refactor api_connection to avoid protobuf message construction if tx_buffer is full May 14, 2025

bdraco added 7 commits May 13, 2025 21:04

cleanup

7cf1db1

cleanup

326df57

cleanup

39abbe6

cleanup

0d81306

cleanup

e4caef7

cleanup

4096c94

true

edd7553

bdraco changed the title ~~Refactor api_connection to avoid protobuf message construction if tx_buffer is full~~ Avoid protobuf message construction when tx buffer is full May 14, 2025

cleanup

c8dcebf

bdraco added 6 commits May 13, 2025 22:59

balance

0db37dd

balance

84a84e7

Merge branch 'dev' into cleanup_api

d7311b0

preen

6e95ef0

Merge remote-tracking branch 'upstream/cleanup_api' into cleanup_api

0d1dae1

lint

b3911ef

bdraco commented May 14, 2025

View reviewed changes

esphome/components/api/api_connection.h Outdated Show resolved Hide resolved

bdraco added 3 commits May 13, 2025 23:20

Update esphome/components/api/api_connection.h

4215cc5

no need to loop twice

2f078d4

Merge remote-tracking branch 'upstream/cleanup_api' into cleanup_api

e244b71

bdraco marked this pull request as ready for review May 14, 2025 05:38

bdraco requested a review from OttoWinter as a code owner May 14, 2025 05:38

probot-esphome bot assigned OttoWinter May 14, 2025

bdraco commented May 14, 2025

View reviewed changes

esphome/components/api/api_connection.cpp Show resolved Hide resolved

bdraco commented May 14, 2025

View reviewed changes

esphome/components/api/api_connection.cpp Outdated Show resolved Hide resolved

bdraco and others added 2 commits May 14, 2025 17:27

optimize

e2453dd

Update esphome/components/api/api_connection.cpp

866a3a3

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

jesserockz approved these changes May 15, 2025

View reviewed changes

bdraco merged commit bb1f24a into dev May 15, 2025
29 checks passed

bdraco deleted the cleanup_api branch May 15, 2025 02:25

jesserockz added this to the 2025.5.0b4 milestone May 19, 2025

jesserockz added a commit that referenced this pull request May 19, 2025

Avoid protobuf message construction when tx buffer is full (#8787)

cab6edd

Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

jesserockz mentioned this pull request May 19, 2025

2025.5.0b4 #8845

Merged

jesserockz added the cherry-picked label May 19, 2025

jesserockz mentioned this pull request May 21, 2025

2025.5.0 #8863

Merged

sa-crespo pushed a commit to sa-crespo/esphome that referenced this pull request May 26, 2025

Avoid protobuf message construction when tx buffer is full (esphome#8787

22bb9ea

) Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

sa-crespo pushed a commit to sa-crespo/esphome that referenced this pull request May 26, 2025

Avoid protobuf message construction when tx buffer is full (esphome#8787

34562e0

) Co-authored-by: Jesse Hills <3060199+jesserockz@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Avoid protobuf message construction when tx buffer is full #8787

Avoid protobuf message construction when tx buffer is full #8787

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Avoid protobuf message construction when tx buffer is full #8787

Avoid protobuf message construction when tx buffer is full #8787

Uh oh!

Conversation

Uh oh!

What does this implement/fix?

Testing

Types of changes

Test Environment

Example entry for config.yaml:

Checklist:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Example entry for `config.yaml`: