feat(playout): high availablility playout #3149
Conversation
… at rabbitmq's suggestion
…ue will be pruned immediately after, and this is already done on startup. If this line is left in, in an HA environment, if one restarts, all instances clear their queues and repopulate them.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3149 +/- ##
===========================================
- Coverage 70.36% 47.73% -22.64%
===========================================
Files 149 22 -127
Lines 4033 1349 -2684
===========================================
- Hits 2838 644 -2194
+ Misses 1195 705 -490
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
I'm not sure if we should link to something more concrete than the architecture docs? I get why you have - it makes it independent of the installation method, but maybe add some examples for Docker (I'm assuming that's how you're running this.). It's definitely easier to document than the installer approach.
|
My thought process was that if you're wanting to do HA, then you are a much more advanced user. And more concrete details are going to make it harder to figure out what exactly needs to happen. If you are doing HA, a lot of times it's not just a simple docker install. In my case I'm going to be running 3 playout blocks. 2 of them are going to be bare metal installs on a VM doing system ALSA playout (We'll see if I run into #3125, if I do I'll fix that one). Additionally it's hard to give a one size fits all for HA stuff as everyone's requirements are different. I was just trying to provide general instructions and information for how to make it happen. Any specifics that are given are not likely to be useful for anybody else. I linked to the architecture docs because it was the single most useful thing for trying to figure out how to do HA. Even then it left more questions than answers, and I had to spend a few days in the code to feel like I knew what was going on. If you still think specifics are needed, I can put something together with the docker install, but I'm worried it will make things less clear. Your call though. |
|
I really like your experiment, but could we put all those behind feature flags, e.g. a flag in the config file or using en vars. I believe there will be more to change to support all features in HA, so I'd go with an incremental approach and make sure we don't break the existing. For example, playout is also sending information back to the API, which this PR does not handle yet. EDIT: I agree that a solid system should be able to scale from 1 to N without feature flag, but I'd prefer to not break productive systems, especially since the test in libretime are not covering enough to trust those changes. |
|
What else is missing for HA? I would like to turn this from experiment into production ready. Only thing I saw for playout contacting API was notifying liquid soap was ready and downloading show schedule and files, which in this scenario is fine. I will look for other instances. As for notifying about tracks starting, they should all be doing that about the same time, so I don't think notifications needs any changes. Am I missing anything else? |
Description
Allow playout block to be duplicated for high availability.
This is a new feature:
Yes
I have updated the documentation to reflect these changes:
Yes
Testing Notes
What I did:
I configured a show, started up 3 playout blocks, each providing one stream to Icecast. I then verified that I could hear each audio output in Icecast and verified that each one responded correctly to events sent from LibreTime (show cancel, next track, editing the schedule). I then verified I could start and stop any one of the playout blocks without interrupting either of the other streams.
How you can replicate my testing:
See above.