Description
I am facing below issue while adding master node to the cluster.
2020-02-22 15:06:42 DEBUG RedisStateMachine:81 - Decoded AsyncCommand [type=MIGRATE, output=StatusOutput [output=null, error='CLUSTERDOWN The cluster is down'], commandType=com.lambdaworks.redis.protocol.Command], empty stack: true
2020-02-22 15:06:42 DEBUG DefaultEndpoint:76 - [epid=0x26, channel=0x7b387fde, /10.1.2.38:48108 -> ] write() done
[ERROR] [02/22/2020 15:06:42.108] [default-akka.actor.default-dispatcher-6] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a/setSlotAssignmentWorker/migrateSlotKeysWorker] Error migrating slot com.adendamedia.cornucopia.redis.ClusterOperations$MigrateSlotKeysClusterDownException: CLUSTERDOWN
[ERROR] [02/22/2020 15:06:42.111] [default-akka.actor.default-dispatcher-8] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a/setSlotAssignmentWorker/migrateSlotKeysWorker] Command failed: MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,11272,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
com.adendamedia.cornucopia.CornucopiaException$MigrateSlotsException: Command failed: MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,11272,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
at com.adendamedia.cornucopia.actors.MigrateSlotKeysWorker$$anonfun$com$adendamedia$cornucopia$actors$MigrateSlotKeysWorker$$working$1.applyOrElse(MigrateSlots.scala:518)
at akka.actor.Actor$class.aroundReceive(Actor.scala:497)
at com.adendamedia.cornucopia.actors.MigrateSlotKeysWorker.aroundReceive(MigrateSlots.scala:487)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [02/22/2020 15:06:42.128] [default-akka.actor.default-dispatcher-12] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-1b8264e7-7612-4256-9012-4d1f042ce79d] Migrate slot job failed for job MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,12061,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
com.adendamedia.cornucopia.CornucopiaException$FailedSlotMigrationJobException: Migrate slot job failed for job MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,12061,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
at com.adendamedia.cornucopia.actors.MigrateSlotWorker$$anonfun$migratingSlot$1.applyOrElse(MigrateSlots.scala:413)
at akka.actor.Actor$class.aroundReceive(Actor.scala:497)
at com.adendamedia.cornucopia.actors.MigrateSlotWorker.aroundReceive(MigrateSlots.scala:355)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:374)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:511)
at akka.actor.ActorCell.invoke(ActorCell.scala:494)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [02/22/2020 15:06:42.128] [default-akka.actor.default-dispatcher-12] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a] Migrate slot job failed for job MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,11272,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
com.adendamedia.cornucopia.CornucopiaException$FailedSlotMigrationJobException: Migrate slot job failed for job MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,11272,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
at com.adendamedia.cornucopia.actors.MigrateSlotWorker$$anonfun$migratingSlot$1.applyOrElse(MigrateSlots.scala:413)
at akka.actor.Actor$class.aroundReceive(Actor.scala:497)
at com.adendamedia.cornucopia.actors.MigrateSlotWorker.aroundReceive(MigrateSlots.scala:355)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:374)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:511)
at akka.actor.ActorCell.invoke(ActorCell.scala:494)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [02/22/2020 15:06:42.132] [default-akka.actor.default-dispatcher-12] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager] Migrate slot job has failed: MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,12061,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
[ERROR] [02/22/2020 15:06:42.132] [default-akka.actor.default-dispatcher-12] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager] Migrate slot job has failed: MigrateSlotJob(c6210d50abb18bdcec12a53e8fa5cb14987795d8,68f4ec7a7bb70fd27646653457b5b2a916ac995c,11272,Map(68f4ec7a7bb70fd27646653457b5b2a916ac995c -> com.adendamedia.salad.SaladClusterAPI@2413ce93, 872285a811a7ddd74cff86d0c6c12cd90fe99c39 -> com.adendamedia.salad.SaladClusterAPI@5dc668d8, c6210d50abb18bdcec12a53e8fa5cb14987795d8 -> com.adendamedia.salad.SaladClusterAPI@6bbce717, f3c4c56c44ab18c7d7eded5ca938041598a38549 -> com.adendamedia.salad.SaladClusterAPI@2a064ea3),Some(RedisURI [host='10.1.2.39', port=6379]))
[INFO] [02/22/2020 15:06:42.143] [default-akka.actor.default-dispatcher-8] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a] Worker was restarted at path akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a: Give me another job!
[INFO] [02/22/2020 15:06:42.146] [default-akka.actor.default-dispatcher-9] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a] Worker was restarted at path akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a: Give me another job!
[INFO] [02/22/2020 15:06:42.153] [default-akka.actor.default-dispatcher-13] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-1b8264e7-7612-4256-9012-4d1f042ce79d] Worker was restarted at path akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-1b8264e7-7612-4256-9012-4d1f042ce79d: Give me another job!
[INFO] [02/22/2020 15:06:42.161] [default-akka.actor.default-dispatcher-29] [akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-1b8264e7-7612-4256-9012-4d1f042ce79d] Worker was restarted at path akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-1b8264e7-7612-4256-9012-4d1f042ce79d: Give me another job!
[ERROR] [02/22/2020 15:06:42.161] [default-akka.actor.default-dispatcher-4] [akka://default/user/lifecycle] Monitored actor [Actor[akka://default/user/lifecycle/overseer/migrateSlotsSupervisor/migrateSlotsJobManager/migrateSlotWorker-33d30344-102f-4c8d-82a1-cf240fbdbd4a/setSlotAssignmentWorker/migrateSlotKeysWorker#-1278928468]] terminated (akka.actor.DeathPactException)