Over the past few days, the subscriber count on my channel has been fluctuating wildly and inorganically, instantly jumping up and down by up to a hundred. Just look at the plot. Numeric data by minute can be found here. What the heck, YouTube?
So, yesterday I streamed the official YouTube broadcast onto my very own livestream. The motivation was to have a readily available backup VOD immediately after the event, in case the official channel did anything stupid to their version. Turns out this went horribly wrong. The official channel didn’t do anything stupid to their archived stream, and the last four hours were immediately available after the broadcast; some hours later, processing on the complete 7:16:03 version was done and made fully available. Meanwhile, my archive is still being processed after 24 hours, and only the last 1:35:01 is available at the moment; the total length shown in Video Manager also dropped from ~7:17 to 5:50:31 for whatever reason. (As for why mine is still being processed, I suppose YouTube priotizes popular streams.) In short, my backup was unnecessary and a disaster.
But my livestream went further than a backup. I didn’t expect to hit the top in Search, which attracted 5–10% of the total viewership. My stream peaked at ~400 concurrent viewers, compared to maybe ~6000 (or ~8000?) on the official stream. Also, the chatroom was sort of overtaken by Vietnamese fans — unfortunately I didn’t have the slightest clue of what they were talking about; I guess someone shared a link to my stream among them? Anyway, despite my effort to move people over to the official stream, especially after my stream started falling apart (which will be discussed in detail later), some people still stayed and I ended up with >100 at the lowest point.
Time to put on technical gear. Duplicating a YouTube stream is trivial with FFmpeg. A basic one-liner is
where 96 is the format code for 1080p HLS, and xxxx-xxxx-xxxx-xxxx is the secret key available from either the Live Dashboard or ingestion settings for an event with its separate stream.
Realistically, though, I needed to keep a local copy, and I was prepared to restart the job as quickly as possible in case the process died or had to be killed for whatever reason, so here’s the script I used in production:
All went well for about five hours.[1] The latency was below ten seconds. Then the official stream was interrupted for at most a few seconds, and all hell broke loose. The download seems to have picked up just fine, with FFmpeg showing a healthy speed of 1x; I’m not sure if that speed is instantaneous or average — I always thought it’s instantaneous, but when I experienced a similar interruption during the test stream a day earlier (the July 28 warm-up event), the speed gradually climbed from ~0.8x back up to 1x over the course of twenty minutes or so, so the speed may be more complex than just instantaneous, although it doesn’t seem to be a global average as well. I would read the relevant parts of FFmpeg source code if I were really curious. Anyway, whatever that speed means, the RTMP stream simply fell apart. The status on the Live Dashboard begins to cycle through a gray “offline — stream complete”, a red “live — video output low” (technically videoIngestionStarved), and the occasional yellow or green that couldn’t be maintained for more than a couple seconds. Reflected on the served stream, it was buffering all the time. I have no idea why a gap of few seconds could be so disruptive, considering network I/O isn’t remotely at capacity — I would imagine it should be easy to catch up. I would occasionally see a “last message repeated 1 times” message in FFmpeg’s output, but I never saw the actual messages…[2] Maybe the messages were about dropped segments from M3U8 playlists? If that’s the case, and given I/O shouldn’t be a bottleneck, I would guess it’s -re's fault. -re is an input option for reading input at native frame rate; it’s necessary for streaming an archived stream (the first thing I tested), but I doubt its necessity when streaming another livestream — after all, the input can only be read at the native framerate, give or take a little. Whether dropping -re would be stability issues is something to investigate, but there’s a non-zero chance this could explain why FFmpeg just couldn’t catch up after the interruption.
The stream chugged along in the painful cycle for the better part of an hour. Then out of nowhere (segments were still being uploaded normally; I was monitoring my I/O) it got so bad that after three rounds of stopping and buffering the stream did not advance a single bit. I knew it was time to kill the process and accept defeat for my offline recording which was tied to the same process (a restart would apparently cause a short gap in the on-disk version). Somewhat ironically yet predictably, the stream went back to normal immediately. I should have made the hard decision much earlier.
The drama did not end there. YouTube actually expires their HLS feed URL after six hours. Therefore, after my stream went back to normal and my attention shifted back to the official stream (for the results, apparently), at some point FFmpeg just quitted with 403, and went on to fail in an infinite loop (see my script). The Live Dashboard was on my secondary machine by the side and I was only glancing at it from time to time, so I didn’t notice it until maybe twenty seconds later. I removed the expired /tmp/stream_url and kicked off the script again. In hindsight this seemed inevitable for a 7+ hour livestream,[3] but at least the gap could have been shortened if (1) I was refreshing the cached stream URL, say, every hour in the background; (2) I stayed vigilant by issuing a notification whenever ffmpeg quits.
That’s pretty much it. To summarize, here are the lessons I learned:
The -re flag should probably be dropped (make sure to test the output stability before the next production run);
Decouple offline recording from streaming for flexibility. My bandwidth is more than enough to handle two simultaneous downloads;[4]
Be decisive. If the stream can’t keep up, immediately kill and restart the transmission.
Refresh the cached stream URL every hour in the background so that we don’t scramble to fetch a new URL when the six hour expiration mark is hit;
Issue a notification with e.g. terminal-notifier whenever ffmpeg quits.
By the way, as a side effect, the channel’s number of subscribers saw crazy growth on July 29:
I’m currently entertaining the idea of livestreaming Theater performances from live.bilibili.com/48 too, in the future. If I want to do that though, I need to make sure it works completely unattended.
Time and durations are all approximate. I did not keep timestamped logs, and have since removed the botched local copy; in addition, analytics for the livestream isn’t available yet. ↩
I suppose either the messages didn’t end in a newline, or there was an output race condition. ↩
I don’t know a way to feed another input URL, unbeknownst to me in the beginning, to a running ffmpeg process. Maybe there’s a way with ffserver? No idea since I never touched ffserver. ↩
There might be a small cost in terms of latency. ↩
I upload videos to the channel using an adapted version of upload_video.py from YouTube’s API code samples. The script does not provide a progress bar. This is usually not a problem because my connection is pretty fast, and when I upload from my Mac I can always bring up Activity Monitor to check bytes uploaded by the python process. However, when I upload from a shared remote server, sometimes due to spikes in load not under my control, upload would seem slower than usual; so I do want to see a progress bar occasonally.
upload_video.py uses MediaFileUpload which is a blackbox taking only the file path, and when you call next_chunk on an insert request, technically a MediaUploadProgress is returned which tells you the percentage completed, but realistically if you’re streaming the file (you should), the whole file is treated as a single “chunk” (next_chunk is only called once), so it’s pretty much useless.
I looked through the docs and, fortunately, there is a lower level MediaIoBaseUpload class that is less of a blackbox. With this class we can simply hook progress bar callbacks into read and seek of an io.IOBase object. Here’s some skeleton code (where I use an io.BufferedReader wrapped around an io.FileIO object, just like what is done by open):
I’m thrilled to announce that, as of 11:20 EDT, July 15, 2017, the SNH48 Live channel has accumulated one thousand subscribers. A pretty significant milestone. 1000 subscribers and 108k+ views in 115 days.
Also, the channel now shows up on the first page when you search for SNH48 on YouTube:
The channel is listed under popular channels within the SNH48 topic as well:
Of course, the channel’s influence is still miniscule compared to some of the other channels listed in the screenshot above, and I’ll probably never reach the same level of success, since Theater performances simply do not attract the same amount of attention as major television appearances — not even close actually, and my three-hour long VODs simply can’t compete with short videos in terms of average percentaged viewed — an important metric in YouTube’s ranking algorithms. However, I’m quite happy with where I am four months later. I’ll continue striving to be as consistent and professional as possible, and apparently it does pay off.
I want to thank everyone who supported the channel, and hopefully I’ll be motivated enough to see it through to 2k and beyond.
大家好,我是SNH48 Team SⅡ的莫寒。终于到拉票会了,其实当时抽到拉票会的瞬间真的很绝望,因为我觉得我自己要说的都已经说得差不多了,不如就把我的总选宣言循环播放十遍。但是后来发现,大家都是容易心软的人,我再不多说点什么,很害怕你们就被拐走了。所以现在想着以前说“没事啊你们可以去浪啊去开心”的我简直就是个傻子,为什么没有直白一点地说,“我只想你们看着我”呢。
So, why am I starting a blog for snh48live.org? The channel has been up for 113 days since its inception on March 22, 2017, and it now boasts 970 subscribers and 100k+ views. Quite an achievement, and I’m really excited about it. I figured I need a place to record some of the milestones, interesting finds (I’ve never run a YouTube channel seriously prior to this, so I learn new things every once in a while), thoughts and stuff; and since short-form writing has never been my greatest strength, Twitter is not the right platform for this. Hence a blog. Mostly for myself.
I’m not exactly sure what I’m going to write about, other than for the first two or three posts, this one included. The blog may even become dormant after the first week; I just don’t know. Another thing I’m not sure about is the language of this blog. I’ll probably use whatever language I’m most comfortable with on a given topic, which means English for technical stuff, Chinese when I need to speak a lot of proper nouns (girls, performances, etc.), and so on.
I’m choosing Hexo as the generator for this blog, and am using the Even theme as a basis. Hopefully I’ll like Hexo better than Jekyll, which isn’t a high bar to clear.