Huge fluctuations in subscriber count

Over the past few days, the subscriber count on my channel has been fluctuating wildly and inorganically, instantly jumping up and down by up to a hundred. Just look at the plot. Numeric data by minute can be found here. What the heck, YouTube?

目标:整合村内最精华的内容

转发一下自己的一条评论

隔了三天才发剪出的单曲,可以说又后知后觉了。其实总共只花了几十分钟的工作,我来谈下为什么拖到现在吧。

自设立这个频道起,我就一直在考虑一个问题:究竟什么内容适合这个频道?我的初衷自然是上传公演,但做了一段时间之后,我认为这个频道是一个良好的平台,不妨做得更广一点,囊括更多的内容,把它打造成一扇了解SNH48的窗户。于是我加入了《Lucky Seven Baby》、《莫莫有闻》等团综、纪录片以及一些音乐内容。然而新增内容有一个度的问题。比如,既然我上传了《莫莫有闻》(暂且撇开我推莫寒的私心),我是不是应该一视同仁地上传《塞纳河一周播报》、《塞纳河周边研发社》等小节目?上传的小节目、小视频过多自然会稀释频道的核心——公演,使频道的可订阅性下降——我个人不喜欢订阅更新非常频繁的频道,因为会有目不暇接的感觉。这自然不是对其他频道的批判——我尊重也感激其他无差别上传小视频的频道,我们共同组成了对SNH48最完整的覆盖(请容我把自己算进去);只是我有我自己的取舍,想做一个更具统一性的频道而已——我的这方面追求从我的视频封面就可见一斑。我想,我目前的目标简单来说就是整合“村内”最精华的内容;而究竟什么是最精华的内容,究竟哪里是数量和质量的平衡点,这些都比较主观,都是我在摸索的。

回到这次剪出的二十一首单曲。本来没这个打算,但后来考虑到自己看了很多遍莫寒的《梦》(演唱会当时我就私下剪出来了),意识到每一首单曲对不同人群有不同的收藏价值;完整的演唱会虽好,但能直接回顾最喜欢的单曲,有别样的意义。这次,数量之庞大并不能掩盖质量之光华。

记一次失败的转播

众所周知我推莫寒。关于结果并没有什么可说的;江东子弟多才俊,卷土重来未可知,聊以自慰。不论成绩如何,S队还是S队,莫寒还是莫寒,八月五日依旧会开开心心地看公演。

这篇博文我只想记录我转播总选时的一些教训,以供将来借鉴。进入技术话题,切换至英文模式。


So, yesterday I streamed the official YouTube broadcast onto my very own livestream. The motivation was to have a readily available backup VOD immediately after the event, in case the official channel did anything stupid to their version. Turns out this went horribly wrong. The official channel didn’t do anything stupid to their archived stream, and the last four hours were immediately available after the broadcast; some hours later, processing on the complete 7:16:03 version was done and made fully available. Meanwhile, my archive is still being processed after 24 hours, and only the last 1:35:01 is available at the moment; the total length shown in Video Manager also dropped from ~7:17 to 5:50:31 for whatever reason. (As for why mine is still being processed, I suppose YouTube priotizes popular streams.) In short, my backup was unnecessary and a disaster.

But my livestream went further than a backup. I didn’t expect to hit the top in Search, which attracted 5–10% of the total viewership. My stream peaked at ~400 concurrent viewers, compared to maybe ~6000 (or ~8000?) on the official stream. Also, the chatroom was sort of overtaken by Vietnamese fans — unfortunately I didn’t have the slightest clue of what they were talking about; I guess someone shared a link to my stream among them? Anyway, despite my effort to move people over to the official stream, especially after my stream started falling apart (which will be discussed in detail later), some people still stayed and I ended up with >100 at the lowest point.

Time to put on technical gear. Duplicating a YouTube stream is trivial with FFmpeg. A basic one-liner is

1
ffmpeg -re -i "$(youtube-dl -f 96 https://www.youtube.com/user/ChinaSNH48/live)" -bsf:a aac_adtstoasc -c copy -f flv rtmp://a.rtmp.youtube.com/live2/xxxx-xxxx-xxxx-xxxx

where 96 is the format code for 1080p HLS, and xxxx-xxxx-xxxx-xxxx is the secret key available from either the Live Dashboard or ingestion settings for an event with its separate stream.

Realistically, though, I needed to keep a local copy, and I was prepared to restart the job as quickly as possible in case the process died or had to be killed for whatever reason, so here’s the script I used in production:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env zsh
# Source
channel=https://www.youtube.com/user/ChinaSNH48/live
format=96 # 1080p HLS stream
# Destination
endpoint=rtmp://a.rtmp.youtube.com/live2
key=xxxx-xxxx-xxxx-xxxx # Actual key redacted
title='20170729 “我心翱翔”SNH48 Group第四届偶像年度人气总决选演唱会'
# Fetch stream URL
urlfile=/tmp/stream_url # Persist stream URL in between sessions, in case the script needs to be restarted
poll_interval=10 # Interval for polling the live stream when waiting for the stream to begin
stream=
while [[ -z $stream ]]; do
if [[ -f $urlfile ]]; then
stream="$(<$urlfile)"
[[ -z $stream ]] && rm $urlfile
else
stream="$(youtube-dl -g -f $format $channel)"
[[ -n $stream ]] && echo -n $stream >$urlfile || sleep $poll_interval
fi
done
# Stream to YouTube while keeping a local copy
index=0
while :; do
# Do not overwrite any saved segment
while :; do
file="$title $index.ts"
if [[ -f $file ]]; then
(( index++ ))
else
break
fi
done
ffmpeg -re -i $stream -bsf:a aac_adtstoasc -c copy -f flv $endpoint/$key -c copy $file
done

All went well for about five hours.[1] The latency was below ten seconds. Then the official stream was interrupted for at most a few seconds, and all hell broke loose. The download seems to have picked up just fine, with FFmpeg showing a healthy speed of 1x; I’m not sure if that speed is instantaneous or average — I always thought it’s instantaneous, but when I experienced a similar interruption during the test stream a day earlier (the July 28 warm-up event), the speed gradually climbed from ~0.8x back up to 1x over the course of twenty minutes or so, so the speed may be more complex than just instantaneous, although it doesn’t seem to be a global average as well. I would read the relevant parts of FFmpeg source code if I were really curious. Anyway, whatever that speed means, the RTMP stream simply fell apart. The status on the Live Dashboard begins to cycle through a gray “offline — stream complete”, a red “live — video output low” (technically videoIngestionStarved), and the occasional yellow or green that couldn’t be maintained for more than a couple seconds. Reflected on the served stream, it was buffering all the time. I have no idea why a gap of few seconds could be so disruptive, considering network I/O isn’t remotely at capacity — I would imagine it should be easy to catch up. I would occasionally see a “last message repeated 1 times” message in FFmpeg’s output, but I never saw the actual messages…[2] Maybe the messages were about dropped segments from M3U8 playlists? If that’s the case, and given I/O shouldn’t be a bottleneck, I would guess it’s -re's fault. -re is an input option for reading input at native frame rate; it’s necessary for streaming an archived stream (the first thing I tested), but I doubt its necessity when streaming another livestream — after all, the input can only be read at the native framerate, give or take a little. Whether dropping -re would be stability issues is something to investigate, but there’s a non-zero chance this could explain why FFmpeg just couldn’t catch up after the interruption.

The stream chugged along in the painful cycle for the better part of an hour. Then out of nowhere (segments were still being uploaded normally; I was monitoring my I/O) it got so bad that after three rounds of stopping and buffering the stream did not advance a single bit. I knew it was time to kill the process and accept defeat for my offline recording which was tied to the same process (a restart would apparently cause a short gap in the on-disk version). Somewhat ironically yet predictably, the stream went back to normal immediately. I should have made the hard decision much earlier.

The drama did not end there. YouTube actually expires their HLS feed URL after six hours. Therefore, after my stream went back to normal and my attention shifted back to the official stream (for the results, apparently), at some point FFmpeg just quitted with 403, and went on to fail in an infinite loop (see my script). The Live Dashboard was on my secondary machine by the side and I was only glancing at it from time to time, so I didn’t notice it until maybe twenty seconds later. I removed the expired /tmp/stream_url and kicked off the script again. In hindsight this seemed inevitable for a 7+ hour livestream,[3] but at least the gap could have been shortened if (1) I was refreshing the cached stream URL, say, every hour in the background; (2) I stayed vigilant by issuing a notification whenever ffmpeg quits.

That’s pretty much it. To summarize, here are the lessons I learned:

  • The -re flag should probably be dropped (make sure to test the output stability before the next production run);
  • Decouple offline recording from streaming for flexibility. My bandwidth is more than enough to handle two simultaneous downloads;[4]
  • Be decisive. If the stream can’t keep up, immediately kill and restart the transmission.
  • Refresh the cached stream URL every hour in the background so that we don’t scramble to fetch a new URL when the six hour expiration mark is hit;
  • Issue a notification with e.g. terminal-notifier whenever ffmpeg quits.

By the way, as a side effect, the channel’s number of subscribers saw crazy growth on July 29:

Growth in number of subscribers.

I’m currently entertaining the idea of livestreaming Theater performances from live.bilibili.com/48 too, in the future. If I want to do that though, I need to make sure it works completely unattended.


  1. Time and durations are all approximate. I did not keep timestamped logs, and have since removed the botched local copy; in addition, analytics for the livestream isn’t available yet.

  2. I suppose either the messages didn’t end in a newline, or there was an output race condition.

  3. I don’t know a way to feed another input URL, unbeknownst to me in the beginning, to a running ffmpeg process. Maybe there’s a way with ffserver? No idea since I never touched ffserver.

  4. There might be a small cost in terms of latency.

Command line YouTube upload with progress bar

The first technical post.

I upload videos to the channel using an adapted version of upload_video.py from YouTube’s API code samples. The script does not provide a progress bar. This is usually not a problem because my connection is pretty fast, and when I upload from my Mac I can always bring up Activity Monitor to check bytes uploaded by the python process. However, when I upload from a shared remote server, sometimes due to spikes in load not under my control, upload would seem slower than usual; so I do want to see a progress bar occasonally.

upload_video.py uses MediaFileUpload which is a blackbox taking only the file path, and when you call next_chunk on an insert request, technically a MediaUploadProgress is returned which tells you the percentage completed, but realistically if you’re streaming the file (you should), the whole file is treated as a single “chunk” (next_chunk is only called once), so it’s pretty much useless.

I looked through the docs and, fortunately, there is a lower level MediaIoBaseUpload class that is less of a blackbox. With this class we can simply hook progress bar callbacks into read and seek of an io.IOBase object. Here’s some skeleton code (where I use an io.BufferedReader wrapped around an io.FileIO object, just like what is done by open):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import io
import mimetypes
import os
class FileReaderWithProgressBar(io.BufferedReader):
def __init__(self, file):
raw = io.FileIO(file)
buffering = io.DEFAULT_BUFFER_SIZE
try:
block_size = os.fstat(raw.fileno()).st_blksize
except (OSError, AttributeError):
pass
else:
if block_size > 1:
buffering = block_size
super().__init__(raw, buffering)
def seek(self, pos, whence=0):
abspos = super().seek(pos, whence)
# TODO: report position: abspos
return abspos
def read(self, size=-1):
result = super().read(size)
# TODO: report position: self.tell()
return result
class MediaFileUploadWithProgressBar(googleapiclient.http.MediaIoBaseUpload):
def __init__(self, file, mimetype=None, chunksize=googleapiclient.http.DEFAULT_CHUNK_SIZE,
resumable=False):
fp = FileReaderWithProgressBar(file)
if mimetype is None:
mimetype, _ = mimetypes.guess_type(file)
super().__init__(fp, mimetype, chunksize=chunksize, resumable=resumable)

Any progress bar class can be hooked into FileReaderWithProgressBar, with the update method called at places marked as TODO.

See my upload script for a complete implementation (commit 111c200 in case the file is moved or refactored in the future).

1000 subscribers

I’m thrilled to announce that, as of 11:20 EDT, July 15, 2017, the SNH48 Live channel has accumulated one thousand subscribers. A pretty significant milestone. 1000 subscribers and 108k+ views in 115 days.

Also, the channel now shows up on the first page when you search for SNH48 on YouTube:

My video on the first page of search results for “SNH48”.

The channel is listed under popular channels within the SNH48 topic as well:

My channel among “Popular Channels” within the SNH48 topic.

The full page screenshots are here and here.

Of course, the channel’s influence is still miniscule compared to some of the other channels listed in the screenshot above, and I’ll probably never reach the same level of success, since Theater performances simply do not attract the same amount of attention as major television appearances — not even close actually, and my three-hour long VODs simply can’t compete with short videos in terms of average percentaged viewed — an important metric in YouTube’s ranking algorithms. However, I’m quite happy with where I am four months later. I’ll continue striving to be as consistent and professional as possible, and apparently it does pay off.

I want to thank everyone who supported the channel, and hopefully I’ll be motivated enough to see it through to 2k and beyond.

莫寒四选拉票会文字版

莫寒是将才,是S队的中流砥柱(之一),是全团最让我安心的存在。七月十二日,她用行云流水、铿锵有力、酣畅淋漓的十五分钟再次向我们展示了那个隐藏在可爱傲娇受小莫寒深处的,倔强而令人敬畏的灵魂。

点评都是多余的,我只想把她说的话原原本本地记录下来。视频版请点这里


大家好,我是SNH48 Team SⅡ的莫寒。终于到拉票会了,其实当时抽到拉票会的瞬间真的很绝望,因为我觉得我自己要说的都已经说得差不多了,不如就把我的总选宣言循环播放十遍。但是后来发现,大家都是容易心软的人,我再不多说点什么,很害怕你们就被拐走了。所以现在想着以前说“没事啊你们可以去浪啊去开心”的我简直就是个傻子,为什么没有直白一点地说,“我只想你们看着我”呢。

拉票会前期有人来给我留言,说莫莫你拉票会务必好好表现,来弥补什么什么的。我不知道他为什么这么说,但好像觉得我很厉害一样。可是我很无力呀,能仰仗什么呢?感觉前面的差距都不是几千票,而是以万为单位的,就此刻渺小地站在这里的我全力去做了,可能也不存在什么力挽狂澜。我家算是很耿直很心软的吧,中报也特别实诚地all in(笑),得到了第七名。在外人看来可能觉得okay啊,不是达到目标了吗,但是我们自己很清楚啊,还有很多隐藏实力、虎视眈眈,准备在最后一鸣惊人的人。剩下十七天的我,真的太危险了。而且曾经中报都是第五,这次明明票数有所提高,却反而第七,真的特别没有底。我知道大家都很拼很辛苦了,但是如果不能守住现在的一切,就白拼了啊。都付出那么多了,不管过程多绚烂,对于总决选来说,十七天后的结果才是最重要的不是吗?

上一期莫莫有闻有说到面试的事。其实现在来说大多数也是一个形式吧。我是有很现实地想过一些问题的,某些原因说白了其实就是我自我价值不够,处于一个很尴尬的位置。可能公司觉得我的价值在这一层,但外界看来同一层他们能找到更好的人,而低一层的人又更低价,人家又不傻。当然不同层次得到的机会是不一样的,都到了这个高度,怎么甘心去掉落呢。所以需要让大家看到我有这个价值,想让别人知道我可以带来价值。外界的人不会主动来了解你,大概都是公司推荐吧,当然推荐的洗好我也不占便宜。最直观能让外界知道的就是总决选的排名,越往前自然也就越有优势,证明你越有价值。都已经到了这一步,当然要更往前冲啊。有机会去捍卫的东西,在一开始就不能够作出一丁点的让步。

我现在感觉自己变得越来越孤僻了。我是需要从成绩里去获得信心的人,但是现在说句什么都好像是在嘲讽自己的无力。别人在准备下一部戏,别人在筹备下一张EP,别人在等下一个综艺,那我呢,我在干什么(笑)?我好想就只能笑着,然后拼命地捂住耳朵说:我不要知道,我不想知道,但是为什么不是我呢。可是还是得笑着啊,因为我觉得不应该给你们传递这种消极的能量,所以我开直播大都是吐槽啊,吃播啊,嘻嘻哈哈的;如果真的直播给你们哭诉的话,我可能会是第一个崩坏掉的吧。得自己把这些情绪都排解掉,因为在大家面前就会变得脆弱,只有自己的时候,才不得不一个人坚强。但是后来发现,展示脆弱好像更容易得到同情。我现在需要这一份关心,需要这一份支持。你们也多看看我,其实我一点都不好。

(嗯)人一旦空闲起来就很容易想很多,经常做噩梦啊,有可能就是大半夜地惊醒,然后很难过。感觉应该去看看医生,但是觉得光是要给医生解释我所处的境地就很困难,万一他还不能理解,只会让自己更加抑郁吧。做娜娜那集莫莫有闻的那天,一大早来剧场排新公演,然后收到妈妈的信息,说家里有点事(哽咽)。瞬间就很恍惚,但是还是有排练啊,有直播啊,一切都得正常运转不是吗。所以我很正常。下午的时候对完了直播的流程再回来排练,就感觉整个人像僵尸一样,大家在旁边说话都像是天外传来的声音,再多戳我一下我就要爆炸了。所以排到中途其实我就自己下楼了,得跳脱出来冷静。不可以哭。嗯,还有要做的事情。直播完莫莫有闻回去,十点,给我妈打电话,结果发现她手机关机。天知道那个时候我有多害怕。抖着给我爸打电话,才知道我妈是去北京学习,然后现在赶回家,在飞机上,所以关机了。我说,那我明天回来吧。我爸说你别回来了,你们不是马上就要新公演了吗,回来也帮不上什么忙(哽咽,流泪),你就自己忙自己的事情。然后我挂了电话,就抽了自己一巴掌。

第二天呢,继续去剧场排练。无尽的世界当中不是有一段大家都走掉了,就剩我一个人在台上的地方吗。那个时候说这里的意义是我代表着一种信念。和演出的时候不一样,演出台下是有人的啊,但是彩排的时候就没有了。就真的大家都走掉了。一个一个的。台下也是空荡荡的。就剩你一个人坐在那里。灯光黄黄的一点都不刺眼,但是却好像一点一点地把我吞噬,让我沦陷。我那个时候坐在那边低着头,特别特别难过,就是,啊,看吧,又剩你了。大家都走掉了就剩你了吧。一个人什么玩意儿呢。好想消失啊。去你的什么信念。今天失去的东西有什么意义吗?为什么不在还在的时候就好好去珍惜呢?重要的你喜欢的不就应该现在去支持吗?但所有的一切都是一瞬间,抬头的时候就又要笑着啊。没有关系的。

当一切都完成之后,就觉得自己很可怕。害怕被大家发现我是这么可怕的人,是这种一点都不厉害,特别软弱的人。感觉自己像一只猫,因为听说猫咪在要死掉之前,就会默默地离开,找一个隐蔽之处,然后静悄悄地死掉。可是我不想死掉。我一点都不想。

在蛮长时间的空窗期之后终于等来了我的莫莫有闻。我真的非常地感谢帮我策划这档节目的所有的staff,没有让我被剩在中心,赐予了我这一份工作。其实在节目出来之前,都不大被人看好吧。可对我来说是唯一的所以重要的,因而我自己亲身参与了策划,坚持每一期的内容都经我手,直播的流程也都提前自己写,很真心实意地去做,终于也让这个节目得到了大家的认可。有人说,莫寒这个节目很好,但是对她没什么用,不吸粉,还容易让自己的粉丝分票(笑)。这我知道啊,我也很害怕啊,但是我更希望透过这个,可以让更多的人看到,莫寒的特别,莫寒是不可或缺的,莫寒有她必要存在的意义。还有人说谁能把莫寒聊哭吗,我觉得很难吧(笑)。对我来说,看着她们,我是一个过来人,我经历过,所以我能够理解。而正是因为我经历的都过去了,所以我很少再去提及。但是我从未一帆风顺过,直到现在也充满坎坷。我也需要去被珍惜,我也想要被拯救。

所以我真的很羡慕啊,如果不用投也能有很好的资源,那我不用太在意总决选的,还不如大家平时多做点应援,最多就是争口气嘛。但我没有这种运气。我只能靠自己、只能靠你们。一直一直都是这么过来的,都已经拿到的怎么可以再拱手让人。一直被别人嘲笑不给力,为什么不反击?为什么呢?现在不就是应激的反击的时候吗?用最后的成绩去让别人闭嘴就好啦。有成绩才有底气去要求、才有底气去冲击啊。我最害怕的事情就是今年结束了之后:看吧,你们都是垃圾,什么都不给你就是正确的啊。呵,仿佛从一开始就变得毫无疑问,就是如此了。

看了中报,甚至连我都觉得,可能最后要被屠榜了。真的特别的厉害。但是现在还没有啊。我不想、不希望、不愿意,我就是想要挤进去,明明就有的可能性怎么能够让它溜掉呢?即使知道很难,但是不管结局,毅然决然地发起冲击,不就是偶像为了梦想该拥有的勇气吗?所以我需要每一个人的帮助,需要每一个人的支持,每一个人。结局还未定,而且正在由你和我共同地书写,如果已经到最后了就改变不了任何事情了。但是现在还没有到最后呢。我们还有时间,我们还可以去冲,那么就可以改变一切的。有你,有我,有我们每一个人。

请给莫寒投票吧。真的,她真的很需要你们的帮助,很需要你们的支持(音乐起)。我真的,很需要你们的拯救。

(演讲终)

附演唱部分,翻唱林俊杰、五月天《黑暗骑士》:

黑暗里谁还不睡 黑色的心情和斗篷假面
黑夜的黑不是最黑 而在于贪婪找不到底线

脚下是卑微的街 我孤独站在城市天际线
别问我恶类或善类 我只是渴望飞的哺乳类

善恶的分界 不是对立面
而是每个人 那最后纯洁的防线 都逃不过考验

有没有一种考验 有没有一次淬炼
拯救了世界就像 英雄 电影 情节
有没有一种信念 有没有一句誓言
呼唤黎明的出现 yeah yeah~ yeah yeah~
呼唤黎明的出现 yeah yeah~ yeah yeah~
呼唤黎明的出现

越来越大的企业 越来越小的公园
越来越深的幻灭 英雄 电影 情节
面具下的人是谁 或者说不管是谁 wow~
都无法全身而退 yeah yeah~ yeah yeah~
都无法全身而退 yeah yeah~ yeah yeah~
都无法全身而退

(转身,背对观众)

泪水洗去铅华雨露
试图冲击独大局面
尚存坚定前进信念
你们的光点聚成英雄的支点

(转身,面对观众)

当我们都走上街 当我们怀抱信念
当我们亲身扮演 英雄 电影 情节
你就是一种信念 你就是一句誓言 wow~
世界正等你出现 yeah yeah~ yeah yeah~
世界正等你出现 yeah yeah~ yeah yeah~

我正在等你出现
来拯救我

这段是莫寒口述的原创部分,有几个词听得不是很清楚,记录可能有误。


莫寒,2017年7月12日23时11分于微博:

我正在等你出现,
来拯救我。

Hello, world!

This is the mandatory hello world post.

So, why am I starting a blog for snh48live.org? The channel has been up for 113 days since its inception on March 22, 2017, and it now boasts 970 subscribers and 100k+ views. Quite an achievement, and I’m really excited about it. I figured I need a place to record some of the milestones, interesting finds (I’ve never run a YouTube channel seriously prior to this, so I learn new things every once in a while), thoughts and stuff; and since short-form writing has never been my greatest strength, Twitter is not the right platform for this. Hence a blog. Mostly for myself.

I’m not exactly sure what I’m going to write about, other than for the first two or three posts, this one included. The blog may even become dormant after the first week; I just don’t know. Another thing I’m not sure about is the language of this blog. I’ll probably use whatever language I’m most comfortable with on a given topic, which means English for technical stuff, Chinese when I need to speak a lot of proper nouns (girls, performances, etc.), and so on.

I’m choosing Hexo as the generator for this blog, and am using the Even theme as a basis. Hopefully I’ll like Hexo better than Jekyll, which isn’t a high bar to clear.