e.g.:
ffmpeg -y -loglevel verbose -headers "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7? Accept-Language: en-us,en;q=0.5? Accept-Encoding: gzip, deflate? Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8? User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0?" -i https://cdn.provider.url.../content/.../file.m3u8 -tls_verify 1 -c copy -f mp4 "file:resultingfile.mp4"This command would stop at some random point when opening a ".ts" video fragment.
[hls,applehttp @ 0xc16300] Opening 'crypto+https://.../fragment.ts' for readingWaiting for timeouts and playing with the ffmpeg options didn't seem to help.
An strace would indicate the server is not responding, but still is keeping the socket open.
e.g.: (do NOT run strace as a background process with &)
strace ffmpeg -y -loglevel verbose -headers "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7? Accept-Language: en-us,en;q=0.5? Accept-Encoding: gzip, deflate? Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8? User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0?" -i https://cdn.provider.url.../content/.../file.m3u8 -tls_verify 1 -c copy -f mp4 "file:resultingfile.mp4" > /tmp/ffmpeg.log 2>&1
would gives the following output
...
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 5
fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(5, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("...")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=5, events=POLLOUT}], 1, 100) = 1 ([{fd=5, revents=POLLOUT}])
getsockopt(5, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
open("/etc/ssl/certs/ca-certificates.crt", O_RDONLY|O_LARGEFILE) = 6
fstat64(6, {st_mode=S_IFREG|0644, st_size=235192, ...}) = 0
_llseek(6, 0, [0], SEEK_CUR) = 0
fstat64(6, {st_mode=S_IFREG|0644, st_size=235192, ...}) = 0
read(6, "-----BEGIN CERTIFICATE-----\nMIIH"..., 233472) = 233472
read(6, "UMlQMAimTHpKG9n/v55IFDlndmQguLvq"..., 4096) = 1720
read(6, "", 4096) = 0
close(6) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1552330418, tv_nsec=452985119}) = 0
gettimeofday({tv_sec=1552330418, tv_usec=453367}, NULL) = 0
gettimeofday({tv_sec=1552330418, tv_usec=453637}, NULL) = 0
poll([{fd=5, events=POLLOUT}], 1, 100) = 1 ([{fd=5, revents=POLLOUT}])
send(5, "\26\3\1\0\352\1\0\0\346\3\3\\\206\256\373\263ZO\233\356U\344b\204O\3217\275\16fb\331"..., 239, MSG_NOSIGNAL) = 239
poll([{fd=5, events=POLLIN}], 1, 100) = 0 (Timeout)
poll([{fd=5, events=POLLIN}], 1, 100) = 0 (Timeout)
poll([{fd=5, events=POLLIN}], 1, 100) = 0 (Timeout)
...
Perhaps the server was doing some connection throttling?
Workaround: proxy all trafic with tinyproxy
- Install tinyproxy:
sudo apt install tinyproxy
- tweak the following config items: sudo nano /etc/tinyproxy/tinyproxy.conf
# I lowered the timeout from 600s to 60s
Timeout 60
...
# only accept localhost connections
Allow 127.0.0.1
...
#'stealth mode'
DisableViaHeader Yes -
restart the daemon:
sudo systemctl restart tinyproxy
-
a quick test will validate the proxy is working:
wget https://www.google.com -e use_proxy=yes -e http_proxy=127.0.0.1:8888
ffmpeg syntax: (recent ffmpeg versions have a -http_proxy option)
http_proxy="http://127.0.0.1:8888" ffmpeg -y -loglevel verbose -headers "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7? Accept-Language: en-us,en;q=0.5? Accept-Encoding: gzip, deflate? Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8? User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0?" -i https://cdn.provider.url.../content/.../file.m3u8 -tls_verify 1 -c copy -f mp4 "file:resultingfile.mp4"
Youtube-dl syntax:
youtube-dl --proxy http://127.0.0.1:8888/ ...
You can check the tinyproxy log to validate the proxy is working correctly:
sudo tail -f /var/log/tinyproxy/tinyproxy.log
INFO Mar 13 12:34:53 [525]: No upstream proxy for ...
CONNECT Mar 13 12:34:53 [525]: Established connection to host "ondemand-b.lwc.vrtcdn.be" using file descriptor 8.
INFO Mar 13 12:34:53 [525]: Not sending client headers to remote machine
CONNECT Mar 13 12:34:54 [523]: Connect (file descriptor 7): localhost [127.0.0.1]
CONNECT Mar 13 12:34:54 [523]: Request (file descriptor 7): CONNECT ...:443 HTTP/1.1
INFO Mar 13 12:34:54 [523]: No upstream proxy for ...
CONNECT Mar 13 12:34:54 [523]: Established connection to host "..." using file descriptor 8.
INFO Mar 13 12:34:54 [523]: Not sending client headers to remote machine
INFO Mar 13 12:34:54 [525]: Closed connection between local client (fd:7) and remote client (fd:8)
INFO Mar 13 12:34:54 [523]: Closed connection between local client (fd:7) and remote client (fd:8)
