If the stream is unstable, then it waits too long to bounce, forcing a new stream to be created.

[tfabris]

I observed a behavior several times on 2024-04-01 where:

• The stream was somehow interrupted. Don't know why, there was no detected network issue, perhaps it was an error on YouTube's side, one of my API calls indeed said "internal error" so maybe everything was fine on my end and it was YouTube that was on crack that day.
• The stream status, as reported by the YouTube API, was repeatedly healthStatus: noData, streamStatus: inactive.
• I think maybe the code was in a hysteresis loop and giving the stream a chance to come back up on its own without bouncing it.
• However in the new YouTube world since their changes, you can't give it time to come back up on its own. If you wait too long without bouncing the stream, then YouTube will auto-end your video stream.
• When it ends the video stream, you cannot revive it (as described in issue Rewrite code to work around YouTube deprecating persistent/default stream. #69) and you have to create a new one from scratch. The stream status, if you wait too long, goes to recordingStatus: recorded and lifeCycleStatus: complete.
• My code detects the lifeCycleStatus: complete situation. In that state, it correctly creates a new stream. However, I think that if I had simply bounced the stream sooner, before it got into that state, then maybe I wouldn't have needed to create a new stream.

Relevant log entries:

Error 2024-04-01 13:36:53 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 13:36:53 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 13:37:09 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 13:37:09 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 13:37:26 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 13:37:26 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 13:37:42 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 13:37:42 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 13:37:42 CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: complete.
Error 2024-04-01 13:37:42 CrowCam Controller - The recordingStatus is not good. Value retrieved was: recorded.
Error 2024-04-01 13:37:42 CrowCam Controller - The lifeCycleStatus and recordingStatus indicate that it's time to create a new livestream from scratch. Creating a new livestream now.
Info  2024-04-01 13:37:43 CrowCam Controller - Creating new YouTube Live Broadcast.
Info  2024-04-01 13:37:58 CrowCam Controller - New video is live. Video ID: j-KJTEKXjmI Status: active Key: xxxxxxxxxxxxxxxxx.
Info  2024-04-01 13:38:04 CrowCam Controller - Live stream came back up.
Error 2024-04-01 14:10:48 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 14:10:48 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 14:11:04 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 14:11:04 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 14:11:20 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 14:11:20 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 14:11:37 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 14:11:37 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 14:11:37 CrowCam Controller - Bouncing the YouTube stream for 4 seconds, because the stream was unexpectedly down during the quick test in the main code after having run the Test_Network function.
Error 2024-04-01 14:17:25 CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: complete.
Error 2024-04-01 14:17:25 CrowCam Controller - The recordingStatus is not good. Value retrieved was: recorded.
Error 2024-04-01 14:17:25 CrowCam Controller - The lifeCycleStatus and recordingStatus indicate that it's time to create a new livestream from scratch. Creating a new livestream now.
Info  2024-04-01 14:17:26 CrowCam Controller - Creating new YouTube Live Broadcast.
Info  2024-04-01 14:17:44 CrowCam Controller - New video is live. Video ID: IcqrG9obnYE Status: active Key: xxxxxxxxxxxy.
Info  2024-04-01 14:17:52 CrowCam Controller - Live stream came back up.
Error 2024-04-01 14:28:09 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 14:28:09 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 14:28:09 CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: complete.
Error 2024-04-01 14:28:09 CrowCam Controller - The recordingStatus is not good. Value retrieved was: recorded.
Error 2024-04-01 14:28:09 CrowCam Controller - The lifeCycleStatus and recordingStatus indicate that it's time to create a new livestream from scratch. Creating a new livestream now.
Info  2024-04-01 14:28:11 CrowCam Controller - Creating new YouTube Live Broadcast.
Info  2024-04-01 14:28:26 CrowCam Controller - New video is live. Video ID: g1aOS4opMvw Status: active Key: xxxxxxxxxxxxxxxxx.
Info  2024-04-01 14:28:32 CrowCam Controller - Live stream came back up.
Error 2024-04-01 14:33:39 CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
Error 2024-04-01 14:33:39 CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
Error 2024-04-01 14:33:39 CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: complete.
Error 2024-04-01 14:33:39 CrowCam Controller - The recordingStatus is not good. Value retrieved was: recorded.
Error 2024-04-01 14:33:39 CrowCam Controller - The lifeCycleStatus and recordingStatus indicate that it's time to create a new livestream from scratch. Creating a new livestream now.

[tfabris]

With checkin 1f98a91 I have reduced the number of hysteresis loops in Test_Stream from 4 to 2 and reduced the pause between tests from 15 to 7. That means that if the stream is bad, instead of waiting a whole minute to bounce it, it's now only waiting 14 seconds.

This still isn't perfect though, because the program doesn't re-query the stream throughout the entire run of the CrowCam.sh program. Instead the program currently works like this:

1. Do all the timing-related stuff first (sunrise, sunset, hourly quick bounce, etc.).
2. Quick check the stream state. If bad, bounce it (or rebuild it if lifeCycleStatus says so)
3. For five minutes (well, four minutes and change), keep looping and testing the network ping.
4. Check the stream state again. If bad, bounce it (or rebuild it if lifeCycleStatus says so)

Checkin 1f98a91 improves the hysteresis at steps 2 and 4, but still leaves a five minute "hole" during step 3 where the stream isn't being queried at all. A worst-case scenario would look like this:

• Step 2 passes fine, the stream state is checked and it's fine.
• YouTube fscked up the stream just after step 2, but my network was fine (that's the state I saw on April 1st which made me write this bug)
• Then there could be a five minute period where the stream is bad, but it's not being checked. Five minutes might be too long and still make the livestream die, regardless of my hysteresis settings.

Options to fix this:

• Do nothing. Maybe the changes I already made to the hysteresis of Test_Stream fixed it, but even if they didn't, CrowCam still "worked" on April 1st, it just created a new livestream from scratch several times that day when it maybe didn't need to.
• I could place a Test_Stream inside the 5-minute network test loop instead of once before and once after. However. This is something that previously existed already, and I deliberately deleted it way back at issue YouTube stream went down, but did not bounce automatically, on 2019-04-01. #9, and I've got code comments telling me that I deleted it on purpose. Some reasons not to Test_Stream during every network ping test:
  • Eats up much more YouTube API quota.
  • Slows down the network test loop by an unpredictable amount, since the Youtube API calls might not return quickly, making the overall network test loop run much slower than expected (this was my original reason for deleting this)
• Perhaps I could compromise and do Test_Stream just once at the halfway point of the network test loop, rather than every loop. That would reduce my failure surface from 5 minutes to 2.5 minutes, and would only double my API usage instead of increasing it by an order of magnitude.
• Or, without changing the code logic, I could reduce the number of network tests by half, and then just schedule the script to run twice as frequently. That would have a similar outcome as sticking a stream test in the middle of the network test loop. It would consume more YouTube API quota though, because there are some other things in the script which also consume quota besides Test_Stream. i don't think I like this idea as much as the halfway test idea.
• Consider changing the code which creates new streams and see if you can remove the "Auto Stop" flag. If this can be done, then you'll need to add more complex code which ensures that you "stop" the stream with an API call anytime you end a broadcast on purpose, discover a new stream needs to be created, or split the stream on purpose.

To do in the meantime:

• Watch the logs and see if the April 1st error rears its ugly head again, and see if your changes to the Test_Stream hysteresis was enough to fix it.
• See if you can find documentation about exactly how long the data can stop squirting before YouTube kills your livestream out from under you (this may be two different amounts, depending on whether Auto Stop is enabled or not). UPDATE: It's "about one minute" if the Live Broadcast contentDetails.enableAutoStop is set to true. Don't know what happens if it's set to false, might be worth finding out.

[tfabris]

The error seemed to recur on April 14th, so, need to continue investigating this further:

2024-04-14 18:21:10   CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
2024-04-14 18:21:10   CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
2024-04-14 18:21:18   CrowCam Controller - The streamStatus is not active. Value retrieved was: inactive.
2024-04-14 18:21:18   CrowCam Controller - The healthStatus is not good. Value retrieved was: noData.
2024-04-14 18:21:18   CrowCam Controller - Bouncing the YouTube stream for 4 seconds, because the stream was unexpectedly down in the main code after the Test_Stream function.
2024-04-14 18:22:21   CrowCam Controller - The lifeCycleStatus is not good. Value retrieved was: complete.
2024-04-14 18:22:21   CrowCam Controller - The recordingStatus is not good. Value retrieved was: recorded.
2024-04-14 18:22:21   CrowCam Controller - The lifeCycleStatus and recordingStatus indicate that it's time to create a new livestream from scratch. Creating a new livestream now.