Skip to content

fix(logger): count buffer max_bytes in UTF-8 bytes - #8833

Open
bardrop wants to merge 1 commit into
aws-powertools:developfrom
bardrop-oss:fix/logger-buffer-max-bytes
Open

bardrop wants to merge 1 commit into
aws-powertools:developfrom
bardrop-oss:fix/logger-buffer-max-bytes

Conversation

@bardrop

@bardrop bardrop commented Nov 15, 2026

Copy link
Copy Markdown

Issue number: closes #8830

Summary

Changes

LoggerBufferCache and KeyBufferCache sized items with len(str(item)), which counts characters. They now use one small helper, _item_size(), that returns len(str(item).encode("utf-8")). It is used in all three places (the "bigger than the whole buffer" check, add, and remove_oldest), so the running size stays consistent when items are evicted.

No docs change: docs/core/logger.md already says max_bytes is the "maximum size of the log buffer in bytes".

User experience

Before: with max_bytes=1600, logger.debug("é" * 1000) was buffered with no warning, though the record is about 2.2 KB in UTF-8.

After: the record is counted in bytes, so it is treated like any record bigger than the buffer: PowertoolsUserWarning("Cannot add item to the buffer...") and the log is written right away. ASCII-only logs behave the same as before.

Verification

I used an AI assistant (Claude) to help write this change. I ran the tests below myself.

New tests:

  • tests/unit/logger/required_dependencies/test_logger_buffer_cache.py: test_cache_size_counts_utf8_bytes, test_item_larger_than_cache_in_bytes
  • tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py: test_log_record_exceeding_buffer_size_in_bytes (public Logger API only)

On develop (fix reverted, tests kept):

FAILED tests/unit/logger/required_dependencies/test_logger_buffer_cache.py::test_cache_size_counts_utf8_bytes
FAILED tests/unit/logger/required_dependencies/test_logger_buffer_cache.py::test_item_larger_than_cache_in_bytes
FAILED tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py::test_log_record_exceeding_buffer_size_in_bytes
3 failed, 204 passed

With the fix:

$ uv run pytest -q tests/unit/logger tests/functional/logger
207 passed, 7 warnings

Full uv run pytest -q tests/unit tests/functional on this branch: 4056 passed, 73 skipped, 3 failed. The same 3 fail on develop without my change, and they come from test order, not this PR: once tests/functional/metrics/datadog is collected, ddtrace is imported and adds dd.* keys to log records, so test_append_context_keys_handles_empty_dict and test_log_dict_key_strip_nones see extra keys (both pass when tests/functional/logger runs alone). The third, test_datadog_write_to_log_with_env_variable, passes when run alone. On my machine some full runs also hung in the Redis idempotency tests on develop too, so I ran the logger tests separately as well.

ruff format --check and ruff check on the changed files: clean. mypy aws_lambda_powertools/logging/: "Success: no issues found in 17 source files".


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@bardrop
bardrop requested a review from a team as a code owner November 15, 2026 16:00
@boring-cyborg

boring-cyborg Bot commented Nov 15, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 15, 2026
@sonarqubecloud

Copy link
Copy Markdown

@leandrodamascena leandrodamascena added this to the 3.42.0 milestone Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logger size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: LoggerBufferConfig.max_bytes counts characters, not bytes

2 participants