[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Supports create/alter/drop resource group #40188

Merged
merged 15 commits into from
Jan 9, 2023

Conversation

nolouch
Copy link
Member
@nolouch nolouch commented Dec 27, 2022

Signed-off-by: nolouch nolouch@gmail.com

What problem does this PR solve?

Issue Number: ref #38825

Problem Summary:

What is changed and how it works?

supports the create/alter/drop resource group DDL.

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot
Copy link
Member
ti-chi-bot commented Dec 27, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • BornChanger
  • tiancaiamao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot
Copy link
Member

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 27, 2022
@nolouch nolouch force-pushed the resource-group/curd branch from f1ac436 to cf00526 Compare December 27, 2022 06:05
@nolouch nolouch mentioned this pull request Dec 27, 2022
41 tasks
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch nolouch force-pushed the resource-group/curd branch from cf00526 to 3b5676a Compare December 28, 2022 06:51
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 28, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 28, 2022
@nolouch nolouch force-pushed the resource-group/curd branch 3 times, most recently from 04276af to 53f7679 Compare December 28, 2022 09:23
@nolouch nolouch marked this pull request as ready for review December 28, 2022 09:23
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 28, 2022
@nolouch nolouch force-pushed the resource-group/curd branch 3 times, most recently from 37cae60 to 4078bb7 Compare December 28, 2022 10:09
Signed-off-by: nolouch <nolouch@gmail.com>
@@ -680,6 +680,10 @@ func skipUnsupportedDDLJob(job *model.Job) bool {
model.ActionAlterTableAttributes,
model.ActionAlterTablePartitionAttributes:
return true
case model.ActionCreateResourceGroup,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to let br support resource group DDL. Is there any problem to do so?

Copy link
Member Author
@nolouch nolouch Jan 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see placement policy not supported, I just following it. Both of them are look like configuration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not the same. Because placement policy is relate to the cluster topology, so restore the placement rule to a different cluster may be not very useful. But resource group is not, I think it's useful to just restore the resource group settings. BTW, if we want to not ignore the resource group DDL, we should also skip the user bind resource group name attribute to let it in a consistent state, it seems hard to do so.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I skip this check first, and will do more after this PR. record in #40371.

ddl/resource_group.go Show resolved Hide resolved
ddl/resource_group.go Outdated Show resolved Hide resolved
ddl/schematracker/checker.go Outdated Show resolved Hide resolved
domain/infosync/resource_group_manager.go Show resolved Hide resolved
domain/infosync/resource_group_manager.go Outdated Show resolved Hide resolved
@@ -211,6 +211,12 @@ func (b *Builder) ApplyDiff(m *meta.Meta, diff *model.SchemaDiff) ([]int64, erro
return b.applyDropPolicy(diff.SchemaID), nil
case model.ActionAlterPlacementPolicy:
return b.applyAlterPolicy(m, diff)
case model.ActionCreateResourceGroup:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only applyDiff operation, no full schema load operation?

Copy link
Member Author
@nolouch nolouch Jan 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added it in 5bacb0, and will add more test after this huge PR:(. record #40370

go.mod Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 31, 2022
ddl/ddl_api.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 5, 2023
Signed-off-by: nolouch <nolouch@gmail.com>
Copy link
Contributor
@BornChanger BornChanger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 6, 2023
@BornChanger
Copy link
Contributor

/retest

Signed-off-by: nolouch <nolouch@gmail.com>
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch nolouch force-pushed the resource-group/curd branch from 491fcf8 to 112c804 Compare January 6, 2023 09:14
Signed-off-by: nolouch <nolouch@gmail.com>
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 6, 2023
Copy link
Contributor
@glorv glorv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @tiancaiamao PTAL again, thanks

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 7, 2023
@tiancaiamao
Copy link
Contributor

Please re-generate the parser.go to resolve conflicts @nolouch

Signed-off-by: nolouch <nolouch@gmail.com>
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 9, 2023
@glorv
Copy link
Contributor
glorv commented Jan 9, 2023

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: a253054

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 9, 2023
@glorv
Copy link
Contributor
glorv commented Jan 9, 2023

/test check-dev

@nolouch
Copy link
Member Author
nolouch commented Jan 9, 2023

/test unit-test

@ti-chi-bot
Copy link
Member

@nolouch: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit b477b1c into pingcap:master Jan 9, 2023
@nolouch nolouch deleted the resource-group/curd branch January 9, 2023 06:53
ti-chi-bot added a commit that referenced this pull request Jan 12, 2023
* br: add retry for azblob read file (#39650)

close #39649

* copr: fix data race in the mppIterator (#39772)

close #39771

* testkit: set GOMAXPROCS in test (#39766)

* statistics: support download history stats from stats_history (#39701)

* *:  Support Failed-Login Tracking and Temporary Account Locking (#39322)

close #38938

* ddl: mark the writes from delete-only and drop them on merge (#39796)

close #39731

* fix: update ioutil func&pruning case (#39798)

* dumpling: meter network usage (#39774)

close #39674

* staleread: fix flaky test TestStaleReadProcessorWithExecutePreparedStmt (#39805)

close #39804

* domain: fix data race in the MockInfoCacheAndLoadInfoSchema (#39802)

close #39801

* *: upgrade rules_go (#39797)

* *: Add backfill job related tables and operations  (#39616)

close #37122

* ttl: implement ttl job schedule framework (#39472)

close #39471

* *: stable tiflash test in ddl (#39825)

close #39831

* planner: fix index prefix matching (#39377)

close #14581

* parser: add cast to array and member of support (#39820)

close #39821

* fix: update ioutil.readall (#39850)

* planner:  add three fields to statement summary table (#39828)

ref #39199

* ddl: fix unexpect fail when create expression index (#39822)

close #39784

* ttl: ouput TTL comment format when show create table (#39853)

close #39852

* dumpling: add enable paging for tidb server (#39791)

close #39790

* planner: support explain format='cost_trace' on model ver2 (#39840)

ref #35240

* ttl, variables: add system variables for ttl job schedule (#39727)

close #39725

* *: fix slow query nil pointer panic when get session for TTL (#39865)

close #39864

* ttl: Add metrics for TTL (#39849)

close #39848

* statistics: support gc outdated historical stats (#39788)

* planner: rename `general plan cache` to `non-prepared plan cache` (#39873)

* planner: rename 'tidb_enable_general_plan_cache' and 'tidb_general_plan_cache_size' to 'tidb_enable_non_prepared_plan_cache' and 'tidb_non_prepared_plan_cache_size'

* planner: binding from cluster statements summary (#39737)

ref #39199

* ttl: modify default value of `tidb_ttl_delete_batch_size` to 100 (#39878)

close #39877

* ddl: close lightning writers after the import is complete (#39879)

close #39800

* store/copr: handle region error from client (#39838)

ref #39361

* ttl: fix scan workers shrink (#39860)

close #39859

* docs: fix typos (#39870)

* domain: support dump sql meta in plan replayer (#39863)

close #39883

* SlowLog: Update `Wait_TS` field in slow log for every sql  (#39843)

close #39713

* session: fix a bug for InitDDLJobTables (#39861)

ref #39854

* ttl: reschedule scan tasks after update task state (#39891)

close #39890

* ttl: TTL support split scan ranges for some primary key with more than one columns (#39896)

close #39895

* *: add dependabot (#39229)

* expression: support `json_overlaps` function (#39867)

ref #39866

* executor, planner: `InitLogger` before `CreateMockStore` in tests to avoid data race (#39900)

close #39892

* statistics: add source column for stats_meta_hisotry table (#39835)

* ttl: fix ttl txn will not return error when rolling back (#39918)

close #39917

* expression: close recordset (#39899)

* br: make more sane variables of ticking timeout (#39887)

close #39620

* expression: support `member of` function (#39880)

ref #39866

* br: fix err overwrite (#39935)

close #39934

* ttl: add log in the whole lifecycle of a job (#39931)

close #39930

* ttl: use optimistic transaction with 0 retry limit for ttl (#39924)

close #39923

* ttl: periodically update state for a job in heartbeat (#39939)

close #39938

* parser: support syntax set binding status by sql digest  (#39515)

ref #39199

* *: lock row keys during merging back from temp index (#39936)

close #39929

* ttl: initialize notifyStateCh in ttl job manager (#39956)

close #39955

* domain, statistics: periodically update stats healthy distribution (#39739)

close #39349

* ttl: enable 1pc for ttl delete (#39958)

close #39957

* docs/design: format the name of 'Adding-Index-Acceleration' RFC. (#39952)

* *: fix auto analyze worker crash when killed by global memory limit (#39946)

close #39947

* doc: add a proposal for foreign key (#35982)

* domain: Support plan replayer continus capture (#39926)

* expression, json: fix cast json as string with shorter length (#39970)

close #39963

* sessionctx, executor: fix failure of reading MaxPreparedStmtCount (#39736)

close #39735

* resourcemanger: create cpu monitor  (#39962)

ref #39657

* *: prevent cursor read from being cancelled by GC (#39950)

close #39447

* dumpling: supports foreign key limitations (#39914)

close #39913

* build(deps): bump golang.org/x/net from 0.2.0 to 0.4.0 (#39996)

* planner: fix pessimistic-auto-commit not working for point plans (#39993)

close #39928

* fix: ebs volume restore failure since incorrect unfinished volumes (#39975)

close #39974

* planner, json: restore cast flag after substitute column in cast (#39997)

close #39806

* staleread, session: internal write request should be accepted with external ts (#39967)

close #39966

* *: revert 38302, forbid modify column on partition table (#39991)

ref #39915, ref #39922

* executor: close recordset again (#40010)

* ddl: retry prepare RPC when meets region error (#39834)

close #39836

* metrics: add EMA cpu usage (#39995)

close #40029

* executor: fix unstable TestMaxPreparedStmtCount (#40024)

ref #39735

* *: support sample for compressed files for adjustment (#39680)

ref #38514

* docs: fix 'enable_global_index' to 'enable-global-index' (#40009)

* expression: introduce `cast(... as ... array)` in expression index (#39992)

close #40045

* planner: fix the issue that NonPrep Cache cannot work with Explain well (#40044)

close #40039

* ddl: fix waiting for wrong schema version if TiDB server restart (#39985)

close #39981

* *: Fix issue 39999, used wrong column id list for checking partitions (#40003)

close #39999

* oomtest: add whitelist for oom test to fix flaky test (#40055)

close #40054

* *: extend skyline pruning diagnostic info and add extra warnings (#39894)

ref #39893

* ddl: add privilege check when alter table add foreign key (#40051)

close #40050

* planner: add more test cases for non-prep plan cache (#40060)

* sessionctx: disable gc aware memory track (#40019)

close #39971

* *: make auto-analyze killable by global memory limit (#39978)

ref #39971, close #39994

* cpu: fix ticker to avoid close early (#40036)

ref #40029

* build(deps): bump golang.org/x/time from 0.2.0 to 0.3.0 (#39912)

* ddl: use latest ts to read record for adding index (#40081)

close #40074

* executor: close recordset again (#40073)

* planner: add more test cases for MPP hints (#39933)

* planner: support set binding status by sql digest (#39517)

ref #39199

* ddl: check the limitation when creating multi-valued index  (#39818)

close #40086

* sysvar: allow modifying 'tidb_allow_remove_auto_inc' when SEM is ON (#40083)

close #38238

* *: fix issue of multi-schema change with foreign key (#40042)

close #40037

* planner: check the `ignore-plan-cache` hint in insert-stmt (#40080)

ref #39717, close #40079

* parser: support keep_order and no_keep_order hint (#39965)

ref #39964

* *: filter particularly errors when `truncateAsWarning` is true (#40078)

* metrics: add metrics for ema cpu metrics and GOGC (#40049)

close #40092

* server: avoid reusing cached stmt ctx on cursor read (#40023)

close #39998

* *: optimize mpp probe (#39932)

close #39686

* statistics: fix the sync load sql did'nt running internally (#40087)

close #39511

* server: support download plan replayer continues capture file (#40085)

* *: support foreign key check/cascade when execute replace into statement (#40069)

close #40068

* ddl: fix issue of add foreign key too slow in big table (#40112)

close #40111

* ddl: fix issue of partition table with foreign key and add more test case (#40122)

* parser: fix cast function restore (#40129)

close #40130

* ddl: set context correctly in the setDDLLabelForDiagnosis (#40090)

close #40099

* planner: move code of generating IndexMergePath to a separated file (#40134)

* statistics: fix incorrect datetime value when loading stats (#40084)

close #39336

* ttl: escape parameters for finish job sql (#40118)

close #40115

* ddl: prohibit change name of partitioning column (#40147)

close #40135

* lightning: disable foreign key checks (#40032)

close #40027

* br: recursive split batch create table when restore failure on batch create table (#39829)

close #39029

* store/copr: don't reset concurrency of analyze req to 2 (#40168)

close #40162

* build(deps): bump golang.org/x/oauth2 from 0.2.0 to 0.3.0 (#40140)

* domain: revise plan replayer process log (#40126)

* expression: impl cast as array function (#40076)

close #40128

* session: fix tidb_enable_gc_aware_memory_track after upgrade (#40173)

ref #39971, close #40174

* restore: Fix prealloc (#40176)

close #40177

* metrics: remove Keep Alive OPM metric (#40142)

close #40143

* util: fix data race in the cpu test (#40154)

close #40125

* executor: move some test into writetest  (#40097)

* planner: fix an unstable explain test case  (#40199)

ref #35240

* ddl: Refine the error message to compatible with MySQL when drop a partition table partition key column (#38740)

close #38739

* executor: TiFlash supports stale read (#40048)

close #40047

* ddl: ignore internal http client goroutine in leak tests (#40190)

close #40183

* ddl: add more foreign key test case (#40052)

close #40189

* *: add support for -initialize-sql-file on first bootstrap (#35625)

close #35624

* *: fix PointGet will return an stale value when `tidb_enable_plan_replayer_capture` is set (#40197)

close #40194

* planner: support converting `json_member_of` to IndexMerge to access MVIndex (#40175)

ref #40191

* planner: record reasons when skipping Plan Cache (#40210)

* session: Do not run telemetry loops when it's disabled in config (#40156)

close #40155

* docs/design: REORGANIZE PARTITION design (#38246)

ref #15000, ref #15002

* ddl: Disable RENAME COLUMN for partitioning columns | tidb-test=pr/2054 (#40151)

close #40150

* dumpling: add foreign key e2e test (#40133)

close #40227

* ttl: make the TTL compatible with dumpling, lightning and BR  (#40216)

close #40215

* ttl, domain: setup a customized session pool with stats collector (#40171)

close #40109

* ddl: Support flashback cluster with ddl history (#40209)

ref #40026

* lightning: add foreign key test (#40103)

close #40226

* planner: refine planner code for disaggregated tiflash mode (#39813)

close #39814

* docs: update tidb roadmap based on v6.5 (#40170)

* planner: remove the unnecessary skip-plan-cache flag in StmtCtx (#40235)

* ttl: fix change status sql argument (#40234)

close #40231

* planner: update the plan cache strategy when expressions with parameters affect null-check (#40218)

close #38205, close #40093

* go.mod: upgrade badger to 20221229114011 (#40236)

close #39237

* executor: write multi-valued index (#40172)

close #40207

* expression: implement `unhex` function pushdown to tiflash (#39898)

ref #5112, ref pingcap/tiflash#5112

* planner: remove the unnecessary skip-plan-cache flag in StmtCtx (#40246)

* restore: set config value to default when failed to get config from tikv (#36051)

close #36053

* ddl: check default value again in updateColumnDefaultValue (#40230)

close #40164

* docs: update roadmap (#40244)

* *: remove the support of the old ddl framework (#39684)

ref #39824

* lightning: check peers write stall when switch-mode is disabled (#40228)

close #40163

* ddl, parser: make generated column and expression index same as MySQL (#39888)

close #39826

* session: make plan replayer capture support prepared stmt (#40167)

close #40161

* expression: implement `regexp_replace` function pushdown (#40267)

close #40266

* planner: supply more test for binding from history (#40242)

ref #39199

* ttl: disable ttl job when recover/flashback table/database/cluster (#40268)

close #40265

* admin: impl admin check index for mv index (#40270)

close #40272

* planner: refactor plan-cache UseCache flag (#40256)

ref #36598

* test: fix data race in cast as array (#40277)

close #40276

* planner: support converting `json_overlaps/contains` to IndexMerge to access MVIndex (#40195)

ref #40191

* autoid_service: add unit test for the package (#40193)

* executor: special handling is required when an "auto id out of range" error occurs in `insert ignore into ... on on duplicate ...` (#39847)

close #38950

* util: gorotinue pool (#39872)

close #38039

* metrics: add metrics for plan replayer and historical stats (#40271)

* sysvar: fix circular dependency in rebuildSysVarCache leading to deadlock (#40283)

close #40240

* ddl: fix flaky test TestGlobalVariablesOnFlashback (#40291)

close #40287

* server, tidb-server: kill auto analyze when gracefully shutting down (#40284)

close #40038

* planner: better coordination between the ignore_plan_cache() binding and plan-cache (#40280)

close #40225

* store/copr: fix build batchCop in disaggregated tiflash mode (#40008)

close #40035

* ddl: persist index info after changing backfill state (#40229)

close #40217

* doc: update flashback doc (#40031)

ref #40026

* ddl, parser: Implement the write-reorg state split task related functions, and the related interfaces of backfill worker (#39982)

close #37123

* planner: support pushing down predicates to memory tables in prepared mode (#40262)

close #39605

* planner: open the partial order prop push down for LIST PARTITION (#40290)

close #40273

* br: fix bootstrap domain (#40307)

close #40281

* br: add errch buf for checkpoint (#40166)

close #40165

* planner: skip plan-cache for prepared queries with `INT in (Decimals...)` (#40312)

close #40224

* statistics: support historical stats dump partition table (#40310)

* *: update badger (#40331)

* expression: enlarge timeout for test (#40332)

* resourcemanger: add cpu scheduler (#39886)

close #39657

* util: use go-deadlock to find deadlock (#40288)

close #40293

* *: support `curdate()` as column's default value  | tidb-test=pr/2057 (#40326)

close #38356

* ttl: don't schedule ttl job when EnableTTLJob is off (#40336)

close #40335

* *: add warnings to slow log and slow query table (#39884)

ref #39893

* planner,infoschema,executor: Add tiflash fine grained shuffle support for hash join and aggregation (#40121)

close #40123

* ddl: fill in original default for extra writable columns in batch insert (#40198)

close #40192

* ddltest: update goleak and enable race (#40357)

close #40358

* planner: let the planner can select IndexMerge on MVIndex without hint (#40347)

ref #40191

* makefile: enable deadlock in the with-real-tikv (#40359)

close #40293

* planner: support more types to use IndexMerge to access MVIndex (#40343)

ref #40191

* log-backup: should set default value `gc.ratio-threshold` after pitr finished (#40141)

close #40185

* domain: let plan replayer dump support recording stats mem status (#40334)

* docs: add TiDB Forum link and adjust link order (#39868)

* ddl: support online create multi-valued index (#40304)

close #40337

* ttl: add `ttl_job_interval` attribute and remove `tidb_ttl_job_interval` variable (#40033)

close #40028

* *: remove the support of the `amending transaction` (#39714)

close #40381

* executor: reset the related session vars for both INSERT and REPLACE (#40354)

close #40351

* *: fix static pruning partition table in disaggregated tiflash mode (#40238)

close #40239

* planner: fix typo (#40367)

* session: fix data race in the LazyTxn.LockKeys (#40350)

close #40355

* build(deps): bump golang.org/x/crypto from 0.1.0 to 0.5.0 (#40379)

* executor: fix data race at the ShowExec (#39817)

close #39816, close #40295

* planner: fix panic during starting tidb-server if creating global binding for partition table (#40402)

* fix

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Supports create/alter/drop resource group (#40188)

ref #38825

* session: fix deadlock when init domain failed (#40409)

close #40408

* executor: support insert ignore/duplicate replace into with unique multi-valued index (#40369)

close #40207

* br: support reset_tiflash after ebs restoration (#40124)

close #40208

* ttl: add ttl goroutine exit log (#40416)

close #40415

* planner, executor: split the range for unsigned pk of partition table when limit is used (#40313)

close #40309

* planner: add nil check when handling empty charset in `setUnionFlen` (#40429)

close #40285

* planner: support keep_order and no_keep_order hint (#40089)

ref #39964

* planner: unify the behavior of prepare/execute limit to mysql (#40360)

ref #40219

* gpool: register gpool into resource manager (#40410)

ref #40412

* ttl: Add `CommandClient` to trigger TTL job manually (#40346)

close #40345

* ttl: make ttl's group in granfana collapse by default (#40444)

* br: reset cloud_admin and root after ebs restoration (#39986)

close #40418

* ddl: support displaying sub-job reorg type in admin show ddl (#40387)

close #40386

* planner: support use IndexMerge to access composite MVIndex (#40426)

ref #40191

* planner: add test cases for keep_order and no_keep_order hints (#40321)

ref #39964

* statistics: change the upper bound of the out-of-range estimation to modify count | tidb-test=pr/2012 (#39011)

close #39008

* docs: update CI build badages (#40324)

close #40325

* planner: update the non-prep cacheable checker (#40446)

close #40296

* expression, planner: exclude concat_ws from aggresive folding constant (#38383)

close #36888

* keyspace: introduce keyspace conf and etcd path (#40269)

ref #40425

* ddl: skip deleting the unmatched index key on merge (#40465)

close #40464

* Merge pingcap/master and pingcap/feature/reorganize-partition

* Updated CODEOWNERS for the feature branch

* ci: add document about some task that will be triggered manually (#40459)

* errno, store: fix ErrPDServerTimeout arguments (#40468)

close #40439

* ttl: fix the TTL job reports error when primary key contains a column with type `ENUM` (#40457)

close #40456

* planner: support using DNF to construct IndexMerge to access MVIndex (#40471)

* fixup

* fixup

* fixup

* executor: add partition table testcase for historical stats (#40453)

* add partition table testcase

* add partition table testcase

* fix lint

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* ttl: forbid creating/altering a table with TTL options when pk contains float/double column (#40487)

* ttl: forbid create/alter a table with TTL options when pk contains float/double column

* format

* update

* update

* update

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* planner: make MVIndex be compatible with the `use/force index` hint (#40495)

* fixup

* fixup

* fixup

* fixup

* lightning: fix codecPDClient (#40496)

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com>

* ddl: let concurrent truncate on the same table depend on the previous one (#40501)

* done

Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>

* refine test

Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>

Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* statistics: fix estimation error when ranges are too many and modify count is large (#40472)

* fix

* fmt

* bazel update

* update test result

Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* executor: fix issue of KEY_COLUMN_USAGE table return wrong result about foreign key column (#40491)

Signed-off-by: crazycs520 <crazycs520@gmail.com>

Signed-off-by: crazycs520 <crazycs520@gmail.com>
Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* sysvar: add switch for plan replayer capture using historical stats (#40492)

* add switch

* fix

* fix

* fix

Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: upgrade go1.19.5 (#40483)

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* br: need init MDL variable when GetDomain (#40512)

Signed-off-by: joccau <zak.zhao@pingcap.com>

Signed-off-by: joccau <zak.zhao@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* ddl: avoid commit conflicts when updating/delete from mysql.tidb_ddl_reorg. (#38738)

* Added test case

* ddl fix #38669.

The issue was that mysql.tidb_ddl_reorg table was updated by an
inner transaction after the outer transaction started,
which then made a commit conflict in the outer transaction,
when it deleted the same row.

* Fixed typo in comment

* Added test case for #24427

* Disabled tests for CI testing

* Revert "Disabled tests for CI testing"

This reverts commit 17c28f3.

* Revert "Revert "Disabled tests for CI testing""

This reverts commit 65c84d9.

* removed test skips

* Clean up the tidb_ddl_reorg entry after DDL is completed

* Use a cleanup job afterwards instead.

* Fixed test

* Moved cleanup before asyncNotify

* More detailed test failure log

* Refined test error message

* Injecting timoeut to get stack traces from CI

* Updated Debug Dump on timeout

* Delete mulitple entries in tidb_ddl_reorg if needed

* Linting

* Linting

* Added CI debug logs

* Linting + CI debugs

* fixed CI debug

* Try to cleanup also if job.State == synced

* check for non-error of runErr instead of error...

* Use a new session, instead of reusing worker.sess

* Also handle case when job == nil

* Removed CI debug logs

* Misssed change session from w.sess to newly created sess

* Improved TestConcurrentDDLSwitch and added CI debug logs

* Always cleaning up all orphan mysql.tidb_ddl_reorg entries

* linting

* Also cleanup if job is nil

* Updated TestModifyColumnReorgInfo + CI debug logs

* more CI debug

* refactored the cleanupDDLReorgHandle code

* Added missing cleanup in handleDDLJobQueue

* Removed debug panic

* Code cleanup

* Test updates

* Debug cleanup

* Cleaned up test after removal of old non-concurrent DDL code merge

* Linting

* always wrap changes to tidb_ddl_reorg in an own transaction

+ fixed some typos

* Minimum fix

* Always update reorg meta, not only on error

* Issue is here :)

* Fixed newReorgHandler

* Wrapped more tidb_ddl_reorg changes into separate transactions

* linting

* Removed updateDDLReorgStartHandle

* cleanups

* Made runInTxn a method on *session, instead of normal function

* Update test

* Final touches

* Removed duplicate test

* CleanupDDLReorgHandles should only be called from HandleJobDone.

* Variable rename

* Renamed 'delete' variabel name

* Updated test

* small revert

* Removed timeout debugging code

* Simplified the cleanup to only start a new txn and not a new session

* Reverted the change of GetDDLInfo

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Post merge fix

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>
Signed-off-by: crazycs520 <crazycs520@gmail.com>
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
Signed-off-by: joccau <zak.zhao@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com>
Co-authored-by: Song Gao <disxiaofei@163.com>
Co-authored-by: lastincisor <64511824+lastincisor@users.noreply.github.com>
Co-authored-by: tangenta <tangenta@126.com>
Co-authored-by: Bin <49082129+songzhibin97@users.noreply.github.com>
Co-authored-by: Obliviate <756541536@qq.com>
Co-authored-by: Lynn <zimu_xia@126.com>
Co-authored-by: YangKeao <yangkeao@chunibyo.icu>
Co-authored-by: Daniël van Eeden <git@myname.nl>
Co-authored-by: xiongjiwei <xiongjiwei1996@outlook.com>
Co-authored-by: fzzf678 <108643977+fzzf678@users.noreply.github.com>
Co-authored-by: Hangjie Mo <mohangjie1995@gmail.com>
Co-authored-by: 王超 <cclcwangchao@hotmail.com>
Co-authored-by: Chunzhu Li <lichunzhu@stu.xjtu.edu.cn>
Co-authored-by: Yuanjia Zhang <zhangyuanjia@pingcap.com>
Co-authored-by: you06 <you1474600@gmail.com>
Co-authored-by: raymonder jin <rayjingithub@gmail.com>
Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com>
Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com>
Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: Yifan Xu <30385241+xuyifangreeneyes@users.noreply.github.com>
Co-authored-by: bb7133 <bb7133@gmail.com>
Co-authored-by: Xiaoju Wu <wuxiaoju@pingcap.com>
Co-authored-by: crazycs <crazycs520@gmail.com>
Co-authored-by: zyguan <zhongyangguan@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: fengou1 <85682690+fengou1@users.noreply.github.com>
Co-authored-by: wjHuang <huangwenjun1997@gmail.com>
Co-authored-by: mychoxin <mychoxin@gmail.com>
Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com>
Co-authored-by: CbcWestwolf <1004626265@qq.com>
Co-authored-by: hackersean <hackersean@users.noreply.github.com>
Co-authored-by: Yujie Xia <xiayjchn@gmail.com>
Co-authored-by: jiyfhust <jiyfhust@163.com>
Co-authored-by: hehechen <chentongli@pingcap.com>
Co-authored-by: Morgan Tocker <mtocker@squareup.com>
Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>
Co-authored-by: guo-shaoge <shaoge1994@163.com>
Co-authored-by: Ran <huangran.alex@gmail.com>
Co-authored-by: baishen <baishen2009@gmail.com>
Co-authored-by: lance6716 <lance6716@gmail.com>
Co-authored-by: xzhangxian1008 <xzhangxian@foxmail.com>
Co-authored-by: tiancaiamao <tiancaiamao@gmail.com>
Co-authored-by: Jk Xu <54522439+Dousir9@users.noreply.github.com>
Co-authored-by: djshow832 <zhangming@pingcap.com>
Co-authored-by: Yiding Cui <winoros@gmail.com>
Co-authored-by: yibin <huyibin@pingcap.com>
Co-authored-by: Zak Zhao <57036248+joccau@users.noreply.github.com>
Co-authored-by: luzizhuo <496521310@qq.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: ShuNing <nolouch@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: wuhuizuo <wuhuizuo@126.com>
Co-authored-by: ystaticy <y_static_y@sina.com>
Co-authored-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants