8000 [BUG] Changelogs not rotating on mfsmaster with CHANGELOG_SAVE_MODE = 0 · Issue #621 · moosefs/moosefs · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[BUG] Changelogs not rotating on mfsmaster with CHANGELOG_SAVE_MODE = 0 #621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
elgatopanzon opened this issue Dec 12, 2024 · 11 comments
Open

[BUG] Changelogs not rotating on mfsmaster with CHANGELOG_SAVE_MODE = 0 #621

elgatopanzon opened this issue Dec 12, 2024 · 11 comments

Comments

@elgatopanzon
Copy link
elgatopanzon commented Dec 12, 2024 •
edited
Loading

Have you read through available documentation, open Github issues and Github Q&A Discussions?

Yes

System information

Your moosefs version and its origin (moosefs.com, packaged by distro, built from source, ...).

4.56.6, moosefs repo

Operating system (distribution) and kernel version.

Debian 12 6.9.7+bpo-amd64

Describe the problem you observed.

I deployed my cluster with CHANGELOG_SAVE_MODE = 0, and noticed that the changelog.0.mfs file continues to grow. Right now, it's 255GB.

This prompted a short investigation into the code to see how rotation is handled. I saw that when change mode is 0, there's a call to the bgsaver changelog_rotate method. When it's not 0, the rotation is performed directly in the meta_term method.

I believe the problem may be directly in bgsaver.c and there is a chance the bgsaver thread never receives the rotate request. I have not confirmed or tested that.

Can you reproduce it? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

  1. Deploy mfsmaster and cluster with CHANGELOG_SAVE_MODE = 0
  2. Write some files and metadata
  3. Stop the server, changelog.0.mfs will be present in the /var/lib/mfs directory along with metadata.mfs (written on exit)
  4. Start the server again
  5. Update config to CHANGELOG_SAVE_MODE = 1
  6. Reload server
  7. Stop server
  8. The file is now changelog.1.mfs
  9. Start server again
  10. A new file is created, changelog.0.mfs
  11. Stop server
  12. The file is now changelog.1.mfs and changelog.2.mfs
  13. Start the server again
  14. Again a new file is created, changelog.0.mfs
  15. Update the config to CHANGELOG_SAVE_MODE = 0
  16. Stop the server
  17. The files are still changelog.0.mfs etc. without rotation
  18. Start server again
  19. Files are the same, it continues writing to changelog.0.mfs indefinitely
  20. Repeat from step 5 to confirm the change back to CHANGELOG_SAVE_MODE = 1 consistently starts rotation again

Note: test was performed in a live cluster environment

Include any warning/errors/backtraces from the system logs.

No matches for meta changelog or data with warn/error state.
@chogata
Copy link
Member
chogata commented Dec 13, 2024

Thank you, we will investigate this issue.

@onlyjob
Copy link
Contributor
onlyjob commented Dec 27, 2024 •
edited
Loading

I also experienced this issue. Metalogger with BACK_LOGS = 25 and BACK_META_KEEP_PREVIOUS = 1 grew changelog_ml.tmp and changelog_ml_back.0.mfs to over 25 GiB each and consumed all remaining free space in the partition.

Master's CHANGELOG_SAVE_MODE is default (unchanged).

On another metalogger instance, changelog_ml.0.mfs exceeded 60 GiB and still growing...

@chogata
8000
Copy link
Member
chogata commented Jan 7, 2025

The issue was located - a fix will be committed soon.

acid-maker added a commit that referenced this issue Jan 8, 2025
@acid-maker
(master) fixed changelog rotation issues (issue #621)
260abbf
@stevew-Purdue
Copy link

Any ideas of when this fix will be available as an Ubuntu 24.04 package in the repository? I'm running v4.56.6 and even setting CHANGELOG_SAVE_MODE to 1 doesn't resolve the issue for me. A restart of mfsmaster will force the change logs to rotate but otherwise the changelog.0 file continues to grow and no rotation takes place. Is there perhaps another way to mitigate the issue until the fix is available in the repository?

Thanks!

@stevew-Purdue
Copy link

I was able to build mfsmaster from the latest source (4.57.1-1) and can confirm that the changelog rotation issue has indeed been fixed. Thanks!

@onlyjob
Copy link
Contributor
onlyjob commented Jan 25, 2025 •
edited
Loading

FYI @stevew-Purdue, "Latest source" 4.57.1 have not been tagged/released ye 8000 t. Perhaps you just compiled head of master branch? Let's not make misleading comments please.

@onlyjob
Copy link
Contributor
onlyjob commented Jan 29, 2025

It would be nice to publish new minor release if only for this change, @chogata.

Commit 260abbf does not apply cleanly on top of 4.56.6 so backporting this change seems to be a bit more difficult than I expected...

@tparker00
Copy link

💯 On a new minor release please. I'm needing to restart regularly to rotate the logs which is a bit of a pain.

@anon314159
Copy link

I can confirm this bug does exist and is reproducible.

@stevew-Purdue
Copy link

By the way, the metalogger has the same problem. It is not rotating its change logs.

@anon314159
Copy link

By the way, the metalogger has the same problem. It is not rotating its change logs.

I can confirm this bug also effects the metalogger service too.

acid-maker added a commit that referenced this issue Feb 26, 2025
@acid-maker
(master) fixed sending changelog rotate packet to metalogger (issue: #…
d604f1c 
…621)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@onlyjob @tparker00 @stevew-Purdue @chogata @elgatopanzon @anon314159
0