8000 Keycloak upgrade challenges · Issue #2671 · vexxhost/atmosphere · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Keycloak upgrade challenges #2671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
fitbeard opened this issue May 20, 2025 · 6 comments
Open

Keycloak upgrade challenges #2671

fitbeard opened this issue May 20, 2025 · 6 comments

Comments

@fitbeard
Copy link
Contributor

I will list here the discoveries I made while configuring high reliability for Keycloak.

A few details about the current configuration:

  • We are using Keycloak with mysql driver/backend
  • We are using PXC which by default do not allows tables without indexes (temporary setting PXC strict mode to PERMISSIVE)

The last nuance made me anxious when testing updates and clean installations of newer versions:)

  1. Both upgrades from 24.x and clean installs directly to 26.1.x and 26.2.x ended with something like this:

WSREP has not yet prepared node for application use [Failed SQL: (1047) DELETE FROM keycloak.USER_CONSENT WHERE ID IN (SELECT ID FROM TEMP_USER_CONSENT_IDS)]

This is message form PXC engine in Keycloak's bootstrap logs. PXC cluster hanged and pods restarted. Only manual internevntion helped restore PXC cluster to normal state.

One thin I noticed is that 26.x.y is using newer liquidbase for migrations. Keycloak =< 25.x is using 4.25.1 and starting 26.0.x liquidbase is switching to 4.29.1

This data can be retireved from table DATABASECHANGELOG

Something is so bad that PXC with PERMISSINE and DISABLED strict mode hangs. Actually dies.
We need to investigate this in a future.

  1. Upgrade from 24.x and clean install directly to 25.x works as expected.
  2. To upgrade and use Keycloak 25.x we need to:
@fitbeard fitbeard mentioned this issue May 20, 2025
Open
@mnaser
Copy link
Member
mnaser commented May 20, 2025

@fitbeard we've seen similar issues when Octavia can also cause weird crashes with PXC through queries. Do you want to try bumping to the latest 8.0 release and see if that's still there?

Otherwise, I'm wondering if it makes sense for us to deploy PostgreSQL and migrate to it.

@fitbeard
Copy link
Contrib 8000 utor Author
fitbeard commented May 20, 2025 •
edited
Loading

@mnaser that's was my second plan to try - bump PXC operator CR and database image.

@fitbeard
Copy link
Contributor Author

@mnaser bad news. With latest 8.0 #2675 situation is the same - crashed PXC.

@mnaser
Copy link
Member
mnaser commented May 27, 2025

@fitbeard this is really stinky, do you have the traceback somewhere that we can report to pxc upstream? seems like crashing a pxc isn't.. ideal

@fitbeard
Copy link
Contributor Author

Low chance of getting an answer but I will try to crash my lab PXC again and collect evidence for a bug report.

@fitbeard
Copy link
Contributor Author

@mnaser your turn: https://perconadev.atlassian.net/browse/PXC-4689

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mnaser @fitbeard
0