Liquidsoap – MPEGTS Stream into RTP over UDP

I finally have the magical unicorn configuration you’ve been looking for. My previous entry showed you how to create an MPEGTS stream via TCP. Here is the configuration for the MPEGTS stream encapsulated into RTP for UDP. This is a single stream with video and audio, no stupid ass SDP file required. This isn’t suggested for clients to connect and use but as an intermediate transport means. Simple instructions on how to get ffmpeg (ffplay) and VLC to access the streams. You may need to consult satanic priests or launch a car into space as a sacrifice for anything beyond these settings.

Obvious settings are host and port. Your encoding schemes are constrained to what the mpegtsmux capabilities are.

#MPEGTS encapsulated to RTP for UDP Streaming.
#Stream can be played by these players.
#ffplay udp://localhost:5000
#vlc rtp://@:5000
output.gstreamer.audio_video(
video_pipeline=
"videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! video/x-h264,profile=baseline ! queue ! muxer.",
audio_pipeline=
"audioconvert ! fdkaacenc bitrate=128000 ! queue ! muxer.",
pipeline=
"mpegtsmux alignment=7 name=muxer ! rtpmp2tpay ! queue ! udpsink host=127.0.0.1 port=5000 sync=true",
source)

Liquidsoap – Gstreamer MPEGTS stream how to

This is a working example using Liquidsoap and it’s Gstreamer output capabilities. You are probably here because there is little to no information on how to configure this other than ambiguous examples with no explanation. I will try to attempt to explain some of the specific configuration options and my experience. First some of the requirements. This article is based on Liquidsoap v1.3.3 and Gstreamer v1.12. You will need several of the plugin packages required for the Gstreamer configuration. I recommend ffmpeg for testing purposes, Gstreamer client commands for testing are beyond painful. This example will require a computer with a CPU with AVX and/or AVX2 extensions. If the CPU does not have these extensions, it is most likely not powerful enough to do the x264 real time encoding. If this is the case you will need to stick with Liquidsoaps built in theora/vorbis video stream at about maximum of 512×376 video dimensions. At the time of this writing, I am not aware of GPU assisted encoding in Gstreamer. You may also need to sacrifice up to four Beef & Bean burritos, unfortunately there is no vegan option at the time of this writing.


#Setup Log Output And Levels For Testing. Once done testing, set to false and 3.
set("log.stdout",true)
set("log.level",10)

#Set Video Frame Width, Height, and Frame Rate. Liquidsoap will resize and if necessary add borders to the videos that are larger or smaller than these dimensions.
#Larger dimensions will increase CPU usage.
#Important: ALL VIDEOS IN SOURCE PLAYLIST MUST BE THE SAME FRAME RATE. If not you will be guaranteed desynchronized audio.
set("frame.video.width",720)
set("frame.video.height",576)
set("frame.video.samplerate",25)
set("gstreamer.add_borders", true)

#Playlist of your video files you want to stream.
#Again I reiterate: ALL VIDEOS IN SOURCE PLAYLIST MUST BE THE SAME FRAME RATE. You are going to be doing a lot of re-encoding.
source = playlist("/home/user/Videos/")

#The output pipeline you've been dreaming for. You can put the knife down now.
#
#video_pipeline: x264enc is the encoder. The default bitrate is 2048kbits, this setting is in kbits. Higher dimensions will require higher bitrate settings.
# tune=zerolatency is required for this setup. We're in the real time baby!
# pass=qual quantizer=20 are Constant Quality encoding and Quality quantizer. Consult a witch doctor for these settings.
# video/x-h264 defines our video stream mime type. profile=baseline is required. Any other profile you are on your own!
#audio_pipeline: fdkaacenc is the encoder. Other examples show the voaacenc. This is old, use the recommended fdkaacenc instead.
# bitrate setting for fdkaacenc is in bytes.
#pipeline: mpegtsmux is the MPEGTS container used for the video/audio stream.
# tcpserversink is the final destination output. The default for host=/port= are localhost:4953 so are not set in this example.
# recover-policy=keyframe sync-method=latest-keyframe These two settings are required. They set proper keyframes for connecting clients,
# otherwise the clients will never sync up as they will never receive a keyframe.
output.gstreamer.audio_video(
video_pipeline="videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! video/x-h264,profile=baseline ! queue ! muxer.",
audio_pipeline="audioconvert ! fdkaacenc bitrate=128000 ! queue ! muxer.",
pipeline="mpegtsmux name=muxer ! tcpserversink recover-policy=keyframe sync-method=latest-keyframe",
mksafe(source))

First you will want to check your LIQ file for errors and then fire up Liquidsoap, if it didn’t end immediately then that is a good sign. The most likely culprit will be not having one of the Gstreamer plugins, which can be easily installed by your package manager.


$ liquidsoap --check why.liq
$

(This is good, anything else your fumble fingers fucked something up)


2018/05/01 17:06:47 [output(dot)gstreamer:5] GStreamer pipeline: appsrc name="video_src" block=true caps="video/x-raw,format=RGBA,width=720,height=576,framerate=25/1,pixel-aspect-ratio=1/1" format=time blocksize=1658880 ! videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! video/x-h264,profile=baseline ! queue ! muxer. appsrc name="audio_src" block=true caps="audio/x-raw,format=S16LE,layout=interleaved,channels=2,rate=44100" format=time ! audioconvert ! fdkaacenc bitrate=128000 ! queue ! muxer. mpegtsmux name=muxer ! tcpserversink recover-policy=keyframe sync-method=latest-keyframe

Look for this output. Any errors in the Gstreamer configuration will occur at this point and exit Liquidsoap. If everything is good and Liquidsoap is happy it will be streaming to the localhost on port 4953. Now let’s see the video stream!


$ ffplay tcp://localhost:4953
Input #0, mpegts, from 'tcp://localhost:4953':0KB sq= 0B f=0/0
Duration: N/A, start: 3614.480000, bitrate: N/A
Program 1
Stream #0:0[0x41]: Video: h264 (Constrained Baseline) (HDMV / 0x564D4448), yuv420p(tv, bt470bg/smpte170m/bt709, progressive), 720x576 [SAR 1:1 DAR 5:4], 25 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:1[0x42](en): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 138 kb/s
3623.20 A-V: -0.030 fd= 3 aq= 22KB vq= 7KB sq= 0B f=0/0

ffplay is part of the ffmpeg package and makes it really easy to test the stream. There should be NO errors. If ffplay is reporting errors the most likely issue is you’ve made changes to the x264enc settings. There is a small margin of error to the conformity of the x264enc settings. The mpegtsmux only supports a handful of video and audio codecs, so be sure to check on what it supports before throwing random encoders at it. It is not recommended that this be used for end clients but as an intermediate type of output. For example, you can use ffmpeg to connect to the stream, transcode and push to an Icecast server.


ffmpeg -re -i tcp://localhost:4953 -f webm -content_type video/webm -c:v libvpx -b:v 1500K -flags:v +global_header -cpu-used 0 -qmin 10 -qmax 42 -deadline realtime -quality realtime -c:a libvorbis -flags:a +global_header icecast://source:yourpassword@localhost:8000/test.webm

Congratulations! You have now entered the foray of alcoholism and drug addiction!

KiwiIRC – How To Properly Start On Boot Under Linux

Ignore the ding dongs telling you to use systemd. It will work, but the pid and log files will not be touched. That’s not very helpful when using the start up script to restart, or reconfig KiwiIRC. Plus it’s handy to have log files to know what the service is doing. Crontab for the win!

Under the user that KiwiIRC is running.

$crontab -e

Then enter this line with the full path to where KiwiIRC is installed.

@reboot /home/user/bin/KiwiIRC/kiwi start

KiwiIRC will now automatically start on boot of the server and the pid and log files will be properly written to.

Icecast Now Playing WordPress Widget Script

Before moving to WordPress from BlogEngineDotNet I had a widget that made a call to a specially created Icecast XSL file to display the current playing track title in a Widget.  This worked great, but it was static and wouldn’t update if the track changed.  Migrating to WordPress, I wanted to achieve the same track playing information but up the game with it updating to display new track information.  The latest Icecast server has built in metadata report in JSON format now so I wanted to use this over parsing an XSL file. This took me roughly about a day and a half to complete, mostly due to nearly all examples of parsing JSON with Javascript do not work and I know little Javascript. Here is an overview of the process invovled.

  • Configure Icecast headers and SSL certificates
  • Insert Javascript in headers.php
  • Insert HTML in Widget for display
Configure Icecast

You can skip this part if you are not using SSL. Also note Apache and Icecast are running on the same server. Since my site defaults to SSL (https) I have to configure Icecast for SSL.

The first step is to create the proper SSL certificate file format that Icecast uses. It requires a Public/Private keypair file. I used my Let’s Encrypt certificates and concatenated them together into one file.

cd /usr/share/icecast
cat /etc/letsencrypt/live/autonarcosis.com/cert.pem > icecast.pem
cat /etc/letsencrypt/live/autonarcosis.com/privkey.pem >> icecast.pem
chown icecast.icecast icecast.pem
chmod go-r icecast.pem

Now to edit the /etc/icecast.xml configuration file to enable an SSL port and point to the icecast.pem file. Create a second listen-socket container with a different port and enable SSL. Insert http-headers container before paths container for access control (this gives permission for the javascript to access the JSON data).  Put the ssl-certificate path setting within the existing paths container.

<listen-socket>
     <port>8002</port>
     <ssl>1</ssl>
</listen-socket>

<http-headers>
    <header name="Access-Control-Allow-Origin" value="*" />
</http-headers>
<paths>
     <ssl-certificate>/usr/share/icecast/icecast.pem</ssl-certificate>
</paths

Save the icecast.xml file. Enable the new port in the firewall.

firewall-cmd --permanent --add-port=8002/tcp
firewall-cmd --reload

Restart Icecast, which will now be listening on an SSL port.

systemctl restart icecast.service

You can view Icecast error log file /var/log/icecast/error.log to see if the SSL certificate loaded properly.  You should see something similar to this.

[2016-03-11  15:41:45] INFO connection/get_ssl_certificate SSL certificate found at /usr/share/icecast/icecast.pem
[2016-03-11  15:41:45] INFO connection/get_ssl_certificate SSL using ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA2
56:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-R
SA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SH
A384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA25
6:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-S
HA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-S
HA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
Configure WordPress

Now it is time to configure WordPress.  Below is the Javascript that I hacked together to pull the Icecast metadata from the data feed in JSON format. This is WordPress modified Javascript and will not work outside WordPress.  Place this Javascript in the HEAD section of your themes headers.php.

<script type="text/javascript">
function radioTitle() {
 
jQuery.ajax({
        url: "https://www.autonarcosis.com:8002/status-json.xsl",
        //force to handle it as text
        dataType: "text",
        async: true,
        success: function(data) {
                        
        //data downloaded so we call parseJSON function 
        //and pass downloaded data
        var json = jQuery.parseJSON(data);
        //now json variable contains data in json format
        //let's display a few items
        // this is the element we're updating that will hold the track title
            jQuery('#track-title').text(json.icestats.source.title);
        // this is the element we're updating that will hold the listeners count
            jQuery('#listeners').text(json.icestats.source.listeners);            
       }
     });
   };
   
jQuery(document).ready(function(){

        setTimeout(function(){radioTitle();}, 2000);
        // we're going to update our html elements / player every 15 seconds
        setInterval(function(){radioTitle();}, 15000); 

});

</script>

Add the HTML to the widget so the information will be displayed.

<p>
</br>
Listeners: <span id="listeners">00</span></br>
Current track: <span id="track-title">LIVE</span>
</p>

That’s all there is to it. There could possibly be a better way or function to do this.

I used this blog entry as a base for figuring this out; https://linge-ma.ws/update-listeners-track-on-a-website-using-icecast-jsonp-and-jquery/

Vanity Top Level Domains – How To Block Them Using Sendmail

Oh the joy! More spam from those stupid ass new top level domains to scam people out of their money with worthless crap. By the time you are done reading this I am sure another TLD will sprout it’s spam wings and flood your mail server with it’s nonsense. Because the world is running out of TLDs so that someone can create hundreds of millions of bogus domains every day to harass us with. Yeah, that’s a good sustainable plan. No, nothing to do with making money.

How do we get rid of this stupid shit? It’s easy with Sendmail.

#cd /etc/mail

Edit the ‘access’ file and add the below list. I’ve sorted to make it easy to add more TLDs as they flood you with spam. This will be a weekly if not daily thing so get used to it. And yes, yes there is a .wang TLD.

# TLD Reject
accountant                REJECT
actor                    REJECT
airforce                REJECT
army                    REJECT
attorney                REJECT
auction                    REJECT
audio                    REJECT
band                    REJECT
blackfriday                REJECT
bid                    REJECT
christmas                REJECT
click                    REJECT
club                    REJECT
consulting                REJECT
cricket                    REJECT
dance                    REJECT
date                    REJECT
degree                    REJECT
democrat                REJECT
dentist                    REJECT
diet                    REJECT
download                REJECT
engineer                REJECT
eu                    REJECT
faith                    REJECT
forsale                    REJECT
futbol                    REJECT
gift                    REJECT
gives                    REJECT
guitars                    REJECT
help                    REJECT
hiphop                    REJECT
hosting                    REJECT
lawyer                    REJECT
loan                    REJECT
link                    REJECT
market                    REJECT
men                    REJECT
mortgage                REJECT
navy                    REJECT
ninja                    REJECT
party                    REJECT
photo                    REJECT
pics                    REJECT
pro                    REJECT
property                REJECT
rehab                    REJECT
republican                REJECT
reviews                    REJECT
review                    REJECT
rip                    REJECT
rocks                    REJECT
science                    REJECT
sexy                    REJECT
show                    REJECT
social                    REJECT
software                REJECT
stream                    REJECT
tattoo                    REJECT
top                    REJECT
trade                    REJECT
us                    REJECT
wang                    REJECT
webcam                    REJECT
win                    REJECT
xyz                    REJECT

Save the file, make the db files and restart Sendmail.

Updated List as of 5/15/17

fail2ban – sendmail-auth – howto

This article is about fail2ban and preventing Sendmail brute force password attacks. Also known as fail2ban’s worthless sendmail-auth configuration.

fail2ban comes with a completely worthless sendmail-auth filter.  It looks for a failure notification from Sendmail that most likely will never happen anymore as previously the brute force attack would make one connection and issue multiple AUTH commands.  This is no longer the case. The brute force attack makes one connection, issues an AUTH command then disconnects and re-connects. This never triggers the Sendmail “possible SMTP attack: command=AUTH” string.

So, what we need to do is something, anything that actually has some sort of real world value, like it actually working? That’d be helpful, right?

#cd /etc/fail2ban/filter.d/
#cp sendmail-auth.conf sendmail-auth.local

Now edit the .local file and replace the worthless regex

failregex = ^%(__prefix_line)s\w{14}: (\S+ )?\[<HOST>\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$

With this

failregex = \[<HOST>\] .*to MTA
            \[<HOST>\] \(may be forged\)
            \[<HOST>\], reject.*\.\.\. Relaying denied

Save the file, then we want to reload fail2ban

# fail2ban-client reload

Sendmail – How To Disable IPv6 When Sending/Relaying

Well, we learned how to configure Sendmail to send to a specific IP address on a per domain basis. Google and it’s business service e-mail is now by default publishing IPv6 addresses for MX records, it’s almost impossible to do a per domain setup.  We do not want to disable IPv6 entirely on the server, but Sendmail keeps sending out via IPv6. How the hell do you make it stop!  It’s quite simple but just a refresher since this is a configuration that is out of sight and mind.  Remember that Sendmail is really two things. A Daemon (the part that listens for incoming mail) and a Client (the part that sends/relays e-mail). Naturally you have the DAEMON settings by default, but not the CLIENT settings.  So here we go.

I have only found one forum post regarding the proper solution to this problem.  It appears not to be properly documented and possibly this is changed behavior in a recent update. However, it does make sense.  Essentially you need to tell the IPv6 stack to use your IPv4 address.

Edit /etc/mail/sendmail.mc

Look for

DAEMON_OPTIONS(`Port=smtp,Addr=xxx.xxx.xxx.xxx, Name=MTA')dnl

Below the above line add this;

CLIENT_OPTIONS(`Family=inet6,Addr=::ffff:xxx.xxx.xxx.xxx')dnl

Save the file, make the db files and restart Sendmail.

Now, Sendmail will use IPv4 for it’s CLIENT operations.

 

 

Sendmail – How To Deliver To IPv4 Address Per Domain

More mail servers are now accepted e-mail via IPv6.  I have had a dynamically assigned IPv6 block on my Comcast Business account for awhile and I have let Sendmail decide what to use, and about 99.9% of mail is delivered via IPv4.  Just recently it appears Comcast has assigned an IPv6 MX record for their mail server. My Sendmail picked this up and now happily attempts to deliver the mail via the IPv6 address.  Unfortunately, it is immediately rejected due to the IPv6 address does not have a PTR record.  Of course Comcast Business is far behind on assigning IPv6 blocks so there is no way to get a static IPv6 block and a PTR entry.

How do I get Sendmail to deliver to the IPv4 address instead?  It’s called the mailertable feature..  You will need this feature enabled in your sendmail.mc file. Most likely it is already enabled.

/etc/mail/sendmail.mc

FEATURE(`mailertable')

 

Now you need to make an entry into the mailertable file with the domain and IPv4 address. In order to get the IPv4 MX address for the domain you can do so by using the host command. We first look up the main domain name to get the MX records. Then lookup the IPv4 address for the MX record.  We now have the IPv4 address to where we want to deliver the mail.

[root@superstar ~]# host comcast.net
comcast.net has address 69.252.80.75
comcast.net mail is handled by 5 mx2.comcast.net.
comcast.net mail is handled by 5 mx1.comcast.net.
[root@superstar ~]# host mx1.comcast.net
mx1.comcast.net has address 96.114.157.80
mx1.comcast.net has IPv6 address 2001:558:fe16:1b::15

 

We now add these lines to our mailertable file.

/etc/mail/mailertable

.comcast.net     esmtp:[96.114.157.80]
comcast.net     esmtp:[96.114.157.80]

 

Don’t forget to issue make to update the db files for Sendmail to see the changes to the mailertable file. And then restart Sendmail.  It will now deliver to the specific IPv4 address.

[root@superstar mail]# make
[root@superstar mail]# service sendmail restart
Redirecting to /bin/systemctl restart  sendmail.service
You bet there is a catch! If the IPv4 address changes, you will need to manually make the change.
That’s it all there is to this. Sendmail is now delivering to the IPv4 address.