This script is a work by Webmaster Seth Leedy, but was inspired by and evolved from the scripts by [email protected]
I noticed that I was mentioned in the Security Now show(epi: 457, 558). Yipee!
Download any or all Steve Gibson’s GRC Security Now podcasts via a bash script.
The script can look at the episodes already downloaded and download the next one.
You can specify the episode(s). Download 1 or a range.
Another function is to search for text within ALL the episodes and copy the episode text to another directory for further reading.
Run it with “-h” for all the other options.
You can submit issues within GitHub or via email to me. Comments below are not actively monitored…
Command line examples:
Will download the latest episode and it will be the TEXT transcriptions.
./GRC-Downloader.sh -eptxt -latest
Will download the latest episode and it will be the PDF transcriptions.
./GRC-Downloader.sh -eppdf -latest
This will download the TEXT and .PDF.
./GRC-Downloader.sh -eppdf -eptxt -latest
The arguments -ahq and -alq is for downloading AUDIO .MP3 files.
-AHQ = High Quality. -ALQ = Low Quality.
The arguments -vhq and -vlq is for downloading VIDEO .MP4 files.
-VHQ = High Quality. -VLQ = Low Quality.
./GRC-Downloader.sh -alq -vlq -eptxt -latest or
./GRC-Downloader.sh -vlq -eptxt -ep 10
You can also download all video, in HD, and text- at once. At 10 downloads a time. *Not all episodes are in HD. The start of the show was not in video, let alone, HD.
Will download every single text copy of the episodes and search for your text (TNO here) and put the results in a special directory for you to open at your leisure.
./GRC-Downloader.sh -dandstxt TNO
Will search only the text episodes already downloaded in the current directory and the cache used in the above -dandstxt option. It will not go online to search.
Copied from the g.securitynow newsgroup hosted on news.grc.com on date 2012-05-12 07:51 AM.
First message. Subject: “Little SN download script”
Hi all,
I wanted to download every episode of Security Now so I could go back and
listen to the back-catalogue (I've only been listening for a few months).
I threw together a little bash script to pull down all the episodes from
grc.com. I'm posting it here in case anyone else wants to use it (you
need wget):
As it states, he was publishing a script to download all Security Now episodes from the server.
This spawned a few messages of how to do it differently.
I am attempting to record all these different ways for future downloaders.
If you attempt this, set aside some space. As of 2012-05-16, approx: 11.8GB for HQ, 3.62GB for LQ.
James Womack:
#!/bin/bash
echo "Download episodes $1 to $2:"
for ((ii=$1;ii<=$2;ii++))
do
printf -v number "%03d" $ii
echo "Episode: $number"
wget http://media.grc.com/sn/sn-$number.mp3
done
Thomas:
Hi James,
I also make use of scripting to maintain my SN-archive. Attached you will find the ones I made. Their function is very simple and as a default (void of argument) will “guesstimate”, i.e. make use of previously downloaded stuff and then based upon that derive “next” episode for download.
Big note: As you, I wrote these in a bit of a haste a few years back and I had absolutely no intention of the code being scrutinized by this community or anyone else for that matter. If you find the code yucky, or beyond, most likely I am already agreeing. If you see solutions in the code that could have been written a lot more clever, again it comes as no surprise at all.
Disclaimer: These scripts are not guaranteed to be failsafe, foolproof or even to work…but to my experience they usually do, at least in line with my initial description. These scripts are definitely not compatible with the intentions of the RIAA, MPAA or any other racketeering organization affiliated with Sony and the other legitimate crooks of our time and society. Then again, anything able to download whatever or for that matter technology in general beyond the abacus is probably incompatible along those lines…
dl_sn_grc.sh:
#!/bin/bash
# Initialization.
declare -i DISK_SPACE
declare -i DISK_SPACE_MIN="5000000"
declare -i EPISODE
NETCAST_URL=
NETCAST_LQ_URL=
NETCAST_TRANSCRIPT_URL=
# Output title.
echo "Security Now Downloader v0.7 (GRC)"
# Check disk space.
DISK_SPACE=$(df -T /data/disk1/ | grep disk1 | awk '{print $5}')
if [ "$DISK_SPACE" -le "$DISK_SPACE_MIN" ]; then
echo "Minimum amount of diskspace not available! Exiting."
exit 1
fi
# Check argument and set episode.
if test "$1"; then
EPISODE="$1"
echo "Episode input: ${EPISODE}"
else
#EPISODE=$(ls -1 ./Audio/*.mp3 | tail -n 1 | grep -io "^sn-..." | grep -o "...$")
EPISODE=$(ls -1 ./Audio/*.mp3 | tail -n 1 | grep -io "[0-9][0-9][0-9]")
EPISODE+=1
echo "Episode input missing, guesstimating: ${EPISODE}"
fi
# Check length.
case "${#EPISODE}" in
1) EPISODE="00${EPISODE}" ;;
2) EPISODE="0${EPISODE}" ;;
esac
# Set episode filename and download.
#EPISODE_NAME="http://dts.podtrac.com/redirect.mp4/twit.mediafly.com/video/sn/sn0${EPISODE}/sn0${EPISODE}_h264b_864x480_500.mp4"
NETCAST_URL="http://media.grc.com/sn/sn-${EPISODE}.mp3"
NETCAST_LQ_URL="http://media.grc.com/sn/sn-${EPISODE}-lq.mp3"
NETCAST_TRANSCRIPT_URL="http://www.grc.com/sn/sn-${EPISODE}.pdf"
echo "Downloading episode ${EPISODE}..."
#wget "$EPISODE_NAME"
echo "...audio ${NETCAST_URL}"
wget -P ./Audio/ "$NETCAST_URL"
echo "...audio (LQ) ${NETCAST_LQ_URL}"
wget -P ./Audio/ "$NETCAST_LQ_URL"
echo "...transcript ${NETCAST_TRANSCRIPT_URL}"
wget -P ./Transcripts/ "$NETCAST_TRANSCRIPT_URL"
echo "...done."
exit 0
dl_sn.sh:
#!/bin/bash
# Initialization.
declare -i DISK_SPACE
declare -i DISK_SPACE_MIN="200000"
declare -i EPISODE
EPISODE_NAME=
# Output title.
echo "Security Now Downloader v0.8"
# Check disk space.
DISK_SPACE=$(df -T /data/disk1/ | grep disk1 | awk '{print $5}')
if [ "$DISK_SPACE" -le "$DISK_SPACE_MIN" ]; then
echo "Minimum amount of diskspace not available! Exiting."
exit 1
fi
# Check argument and set episode.
if test "$1"; then
EPISODE="$1"
echo "Episode input: ${EPISODE}"
else
EPISODE=$(ls -1 *.mp4 | tail -n 1 | grep -io "^sn0..." | grep -o "...$")
EPISODE+=1
echo "Episode input missing, guesstimating: ${EPISODE}"
fi
# Set episode filename and download.
#EPISODE_NAME="http://dts.podtrac.com/redirect.mp4/twit.mediafly.com/video/sn/sn0${EPISODE}/sn0${EPISODE}_h264b_864x480_500.mp4"
EPISODE_NAME="http://twit.cachefly.net/video/sn/sn0${EPISODE}/sn0${EPISODE}_h264b_864x480_500.mp4"
echo "Downloading episode ${EPISODE}..."
wget "$EPISODE_NAME"
echo "...done."
exit 0
Now gdb says to use this
wget http://media.grc.com/sn/sn-{001..352}.mp3
and it spawned a large response thread.
Guy says:
Thanks for the information.
I use Windows – so curl is the tool for the task of topic.
curl -OL http://media.grc.com/sn/sn-[001-352].mp3
Mark Cross adds:
Adding -N check timestamp and wont download existing files (if they haven’t changed) and -c to continue existing downloads:
echo wget -Nc http://media.grc.com/sn/sn-${padding:${#i}}$i.mp3
Update:
A real ‘one liner bash command’ needs some artistic tweaking (for bash 3.1):
a=’printf %03d’ eval “wget -Nc http://media.grc.com/sn/sn-{$($a 1)..$($a 25)}.mp3”
or (won’t mess with the environment vars set on present shell, for bash 3.1):
sh $(beg=1;end=25;a=’printf %03d’;eval “wget -Nc http://media.grc.com/sn/sn-{$($a $beg)..$($a $end)}.mp3”)
and (for bash 4):
sh $(beg=001;end=025;eval “wget -Nc http://media.grc.com/sn/sn-{$beg..$end}.mp3”)
maybe a bit too complex 😉
ObiWan says something interesting for Windows:
Well... if you're running windows, then you may also leverage the BITS
tool (Background Intelligent Transfer System); the critter integrates
with the windows network stack and ensures that ongoing downloads won't
clogger your bandwidth; this means that BITS won't be the faster way to
download stuff but it will, for sure, be the less "impacting" one; you
probably won't even notice that it's fetching stuff
The cornerstone of such a thing would be a tool called "bitsadmin"
http://msdn.microsoft.com/en-us/library/windows/desktop/aa362813%28v=vs.85%29.aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/aa362812%28v=vs.85%29.aspx
http://technet.microsoft.com/en-us/library/cc753856%28v=ws.10%29.aspx
basically it's a cmdline interface to the BITS service; as for every
windows app/service, there are several ways to use it; the most simple
and straightforward one is the following
bitsadmin /transfer JOBX /download /priority normal http://media.grc.com/sn/sn-001.mp3 d:\SN\sn-001.mp3
the above (single line) will create a job called JOBX, add it a file,
that is "sn-001.mp3", tell it to download the file to d:\SN and then,
immediately start the job in sync mode (just like entering "wget..."
from the cmdline) then... ok, there are other ways to use it; for
example, you may create background jobs which will run (and fetch the
desired files) and which, at end (ok or error) will call whatever
script/program or piece of code you want
The BITS subsystem is the one used by windowsupdate to fetch files and
while it isn't "fast" (not its purpose) I think it's worth exploring
ObiWan also came up with this and I think, right now anyways, this would be the best way to do it for Windows users:
The below will fetch all the episode from 001 to 100 (for…) and save
then in c:\sn the episode number is correctly aligned with zeroes by
that “%NBR:~-3%”; basically the code first adds 000 in front of the
number and then picks the rightmost 3 chars
@echo off
if not (%1)==() goto FETCH
for /L %%n IN (1,1,100) DO call %0 %%n
goto QUIT
:FETCH
set NBR=000%1
set NBR=%NBR:~-3%
bitsadmin /transfer J%RANDOM% /download /priority normal http://media.grc.com/sn/sn-%NBR%.mp3 c:\sn\sn-%NBR%.mp3
:QUIT