Description
Hello team, we have uncovered a potential bug on the way the backup is performed on materialized view queries which include regex escaped characters.
We have just discovered this while trying to restore our sentry clickhouse backup in regular restore tests. The backup and restore process using the docker image works perfectly fine, this is only found out at runtime when trying to insert rows, as the materialized view query will fail with an exception such as:
clickhouse-1 | 2025.05.14 02:26:04.643947 [ 268242 ] {4190deb3-cb2e-4f4a-a38c-5d5caac992c0}<Error> DynamicQueryHandler: Code: 36. DB::Exception: The pattern argument is not a valid re2 pattern: missing ): (=|\): while executing 'FUNCTION replaceRegexpAll(k :: 0, '(=|\\)' :: 2, '\\\\1' :: 3) -> replaceRegexpAll(k, '(=|\\)', '\\\\1') String : 5'. (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):
The relevant code is this in sentry, which generates a table schema that looks as follows when it works:
:) DESCRIBE TABLE errors_local
...
tags.key Array(String)
tags.value Array(String)
_tags_hash_map Array(UInt64) MATERIALIZED arrayMap((k, v) -> cityHash64(concat(replaceRegexpAll(k, \'(\\\\=|\\\\\\\\)\', \'\\\\\\\\\\\\1\'), \'=\', v)), tags.key, tags.value)
...
Since 2.6.6, backup restores do this (notice the difference in escaped chars for the replaceRegexpAll). We observe this in more tables that make use of similar escaped regexes:
:) DESCRIBE TABLE errors_local
...
tags.key Array(String)
tags.value Array(String)
_tags_hash_map Array(UInt64) MATERIALIZED arrayMap((k, v) -> cityHash64(concat(replaceRegexpAll(k, \'(=|\\\\)\', \'\\\\\\\\1\'), \'=\', v)), tags.key, tags.value)
...
Reverting our clickhouse-backup image to 2.6.5 solves the issue. We suspect that this change is the cause of the problems:
This might also be a bug or wrong usage on sentry's side by using the escaping incorrectly, our knowledge of clickhouse is not that deep, but it doesn't look like it from the commit linked above.
It's a pretty sneaky behaviour, because people might have bad backups but they won't notice until they try to restore and use the system (the restore itself succeeds).