search
explicitClick to confirm you are 18+

PP Behind the Curtain: Script Shack

Professor PopulistApr 15, 2021, 2:57:51 PM
thumb_up8thumb_downmore_vert

This is the repository for various scripts I use in my writing. If you'd like to use something but don't quite understand it feel free to ask. You can also check out the other #PPBTC for more details.

For all usage examples I will refer to the script files as "filename.sh" so you should replace this with whatever the name of your script is.

Scripts

Book Combine

#!/bin/sh

#Configuration and Variables
project=$1
bookName=$2
projDir=/path/to/folder/containing/project/$project
book=$projDir/$bookName.md

if [ -n "$project" ]; then
	if [ -n "$bookName" ]; then
		echo '#'$bookName > $book
		echo '##Professor Populist' >> $book

		cd $projDir
		fileList=$(ls | grep -v -e '[.]md')

		for f in $fileList
			do
				cat $f >> $book
				echo '' >> $book
			done
	else
		echo 'No book specified'
	fi
else
	echo 'No project specified'
fi

Usage Example:

Type: sh filename.sh LIES LifeInEddiesShadow

Expected Output: The script should create a new file in the project folder with the specified filename and the .md file extension. Book files will be combined in A-Z order.

Create Blog

#!/bin/sh

#Get todays date for filename
newsday=$( date -I )

#Assign blog filename to variable "outfile"
outfile=/path/to/news/folder/DailyNews-$newsday.txt

#Announce filename
echo "Output file: ${outfile}"

#Combine all files in the specified folder into a new file
#Separate each file's text with ========
awk 'FNR==1 && NR>1 { printf("%s\n","========") } 1' /path/to/newsbucket/* > "${outfile}"

#The next two commands remove blank lines from the file.
#This makes it less human readable but more convenient for posting as a Minds blog
cp $outfile /path/to/news/folder/blogTemp.txt
sed -i '/^$/d' /path/to/news/folder/blogTemp.txt

#Announce completion
echo "Finished."

Usage Example:

Type: sh filename.sh

Expected Output: This will create a text file with the name "DailyNews-<TodaysDate>" which contains all the headlines for the day's NewsBrief. It will also create a text file with the name "blogTemp" which will be the same as the other but with all the blank lines removed.

NOTE: Since this script is used for the creation of NewsBriefs and there are 3 a week I have 3 versions of this script, one for each day. I could have also accomplished this using variables in a single script.

Find Clip

#!/bin/sh

#Search clippings for keyword
grep -H $1 /path/to/news/archive/*

Usage Example:

Type: sh filename.sh social.impact

Expected Output: This will output to your terminal window any news clippings containing the words "social impact".

Get Headlines

#!/bin/sh

#List files and ignore our intro text file
FILES=$(ls --hide=1AAAA_INTRO /path/to/newsbucket)

#Get todays date for filename
newsday=$( date -I )

#Assign headline filename to variable "outfile"
outfile=/path/to/news/folder/Headlines-$newsday.txt

#Assign tempfile filename to variable "tempFile"
tempFile=/path/to/news/folder/tempHead

#Change the current working directory to todays newsbucket
cd /path/to/newsbucket

#Add intro to headline file
echo "In this #PPNewsBrief :" >> "${outfile}"

#Get the headline from each newsbucket item
#Add it to the headline file with a blank line in between
for f in $FILES;
	do
		(head -q -n 1 $f; echo '') >> "${outfile}"
	done

#The next two lines remove extra blank lines at the end of the file
head -n -1 $outfile >> $tempFile
mv $tempFile $outfile

Usage Example:

Type: sh filename.sh

Expected Output: This will create a text file with the name "Headlines-<TodaysDate>" which contains all the headlines for the day's NewsBrief.

NOTE: Since this script is used for the creation of NewsBriefs and there are 3 a week I have 3 versions of this script, one for each NewsBrief. I could have also accomplished this using variables in a single script.

Remove Blanks

#!/bin/sh

outfile=$1
#Remove blank lines
cp $outfile /path/to/output/folder/blogTemp.txt
sed -i '/^$/d' /path/to/output/folder/blogTemp.txt

Usage Example:

Type: sh filename.sh '/home/PP/Blogs/MyBlog.txt'

Expected Output: A text file named "BlogTemp" will be created containing the contents of "MyBlog.txt" with the blank lines removed.

Source Outline

 #!/bin/sh

#Set current directory
currentDir=$(pwd)
#Get file list to outline ignoring certain ones
fileList=$(ls | grep -v -e '--' -e '.sh' -e 'TODO')
for f in $fileList
	do
		#Get clips weblink
		fileClips=$(grep -e http $f)
		#Set output filename
		outlineName=$f'--Outline'
		#Set output file location
		outputFile=/path/to/output/folder/$outlineName
		#Write filename to output file
		#Serves as a heading for document
		echo $f > $outputFile

		#Goto News Archive folder
		cd /path/to/news/archive
		for c in $fileClips
			do
				#Find the used clip in the archive
				clipFile=$(grep -l -e $c ./*)
				#Add the clips filename and weblink to outline file
				(echo '#'$clipFile; echo ''; echo $c; echo ''; echo '========') >> $outputFile
		#		echo $c				
			done
		#Return to original directory
		cd $currentDir
	done

Usage Example:

Type: sh filename.sh

Expected Output: This will create a text file with the name "<Filename>--Outline" which contains all the news clippings for each file in the specified folder.

NOTE: This script must be within the project folder to work correctly.

"Special Coverage" Processing

#!/bin/sh

#Set our special coverage sources
BUCKET1FILES=$(find /path/to/special/coverage/bucket/* -type f)
BUCKET2FILES=$(find /path/to/special/coverage/bucket/* -type f)

#Add the proper hashtags
#Remove the headlines from each news item
for f in $BUCKET1FILES;
	do
		(echo '#BBUCKET1'; echo ''; tail -n +2 $f) >> /path/to/news/folder/temp
		mv /path/to/news/folder/temp $f
	done

for f in $BUCKET2FILES;
	do
		(echo '#BUCKET2 #AnotherHashtag'; echo ''; tail -n +2 $f) >> /path/to/news/folder/temp
		mv /path/to/news/folder/temp $f
	done

Usage Example:

Type: sh filename.sh

Expected Output: The script will cycle through the specified folders (in the script code) and remove the headlines and add the specified hashtags.

NOTE: This script modifies the original file so you are advised to use this script on a copy.

Topic Summary (v2) (Single Search Term)

#!/bin/sh

#Set topic from passed argument
topic=$1
#Set project folder location
#Assumes each project has own folder
project=/path/to/project/folder/$2/
#Set todays date for use in filenames
pullDate=$( date -I )

#Make sure project was specified
if [ -n "$2" ]; then 

	#Checks if you want to rebuild the outlines
	if [ -n "$3" ]; then 
		echo 'Rebuilding outlines.....'
		cd $project
		sh aoMulti.sh
	else
		echo 'Using existing outlines'
	fi

	#ClippingSearch
	#Goto News Archive folder
	cd /path/to/news/archive
	#Get list of files
	fileList=$(ls -t)
	#Set output info
	outputFile=/path/to/output/folder/$topic-$pullDate
	outputTOC=/path/to/output/folder/$topic-$pullDate-TOC
	echo 'Searching news clips for '$topic'......'

	for f in $fileList
		do
			#Search files for matching keywords
			pulled=$(grep -i -l -H -e $topic ./$f)

			for i in $pulled
				do
					#Get the clips weblink
					clipLink=$(grep -e http ./$f)
					#Check if clip is already used in project
					dupe=$(grep -e $clipLink $project/*--Outline)
					if [ -n "$dupe" ]; then
						echo 'Ignoring '$f
					else
						#Write to topic file
						(echo '#'$f; echo ''; cat $f; echo ' [Source]'; echo $clipLink; echo '';  echo '========') >> $outputFile
					fi
				done
		done

	#PostsSearch
	#Goto Post Search config folder
	cd /path/to/config/folder
	#Read list of directories to search from file
	#Assumes you keep your posts in multiple folders
	directoryList=$(cat Directories)
	#Set output info
	outputFile=/path/to/output/folder/$topic-$pullDate-POSTS
	outputTOC=/path/to/output/folder/$topic-$pullDate-POSTS-TOC
	echo 'Searching posts for '$topic'......'

	for d in $directoryList
		do
			#Goto Post directory
			cd $d
			#Get list of files
			fileList=$(ls -t)
			for f in $fileList
				do
					#Search files for matching keywords
					pulled=$(grep -i -l -H -e $topic ./$f)
					for i in $pulled
						do
							#Write to topic file
							(echo '#'$f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
						done
				done
		done
	#CommsSearch
	#Goto Comment archive folder
	cd /path/to/comment/archive
	#Get list of files
	fileList=$(ls -t)
	#Set output info
	outputFile=/path/to/output/folder/$topic-$pullDate-COMMS
	outputTOC=/path/to/output/folder/$topic-$pullDate-COMMS-TOC
	echo 'Searching comments for '$topic'......'

	for f in $fileList
		do
			#Search files for matching keywords
			pulled=$(grep -i -l -H -e $topic ./$f)

			for i in $pulled
				do
					#Write to topic file
					(echo '#'$f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
				done
		done

	#PQSearch
	#Goto Quote Search config folder
	cd /path/to/PQs
	#Get list of files
	#Assumes quotes are in multiple directories
	#Get directories from file
	directoryList=$(cat Directories)
	#Set output info
	outputFile=/path/to/output/folder/$topic-$pullDate-PQS
	outputTOC=/path/to/output/folder/$topic-$pullDate-PQS-TOC
	echo 'Searching PQs for '$topic'......'

	for d in $directoryList
		do
			cd $d
			#Get list of files ignoring certain folders we dont want
			fileList=$(ls -t --hide=ISched* --hide=Posted --hide=SchedNext*)
			for f in $fileList
				do
					#Search files for matching keywords
					pulled=$(grep -i -l -H -e $topic ./$f)
					for i in $pulled
						do
							#Write to topic file
							(echo '#'$f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
						done
				done
		done
else
	#Tell user if they did not specify a project
	echo 'No project specified'
fi

Usage Example:

Type: sh filename.sh social.impact LIES

Expected Output: This will create a text file with the name "social.impact-TodaysDate" which contains all the news clippings with text matching "social impact". Other archives (Comments, Posts, Quotes) will have their own results files and have "-COMMS", "-POSTS", and "-PQS" added to the end of the filename. A file with the same naming convention (and appended with "-TOC" will also be created containing the filenames of each matching item.

NOTE: This will check all matching news clips against the already used sources listed in the "--Outline" files of the LIES project folder. To request a rebuild of the outlines before processing instead type: sh filename.sh social.impact LIES rebuild

You will also need to have set up the relevant "Directories" files. I keep my Quotes and my Posts in multiple folders so I add the path to each folder to the "Directories" file (one per line). The script reads this file and scans through each directory provided.

Topic Summary (v2) (Multiple Search Terms)

#!/bin/sh

pullDate=$( date -I )
topicList=$(cat /path/to/topicList)
project=/path/to/project/folder/$1/

if [ -n "$1" ]; then #make sure project specified

	if [ -n "$2" ]; then
		echo 'Rebuilding outlines.....'
		cd $project
		sh aoMulti.sh
	else
		echo 'Using existing outlines'
	fi
	for t in $topicList
		do
			topic=$t
			#ClippingSearch
			cd /path/to/news/archive
			fileList=$(ls -t)
			#fileList=$(grep -i -l -H -e $1 ./* | grep -v -e DailyNews -e Headlines-)
			outputFile=/path/to/output/folder/$topic-$pullDate
			outputTOC=/path/to/output/folder/$topic-$pullDate-TOC
			echo 'Searching news clips for '$topic'......'

			for f in $fileList
				do
					pulled=$(grep -i -l -H -e $topic ./$f)
			#		echo 
					for i in $pulled
						do
							clipLink=$(grep -e ^http ./$f)
							dupe=$(grep -e $clipLink $project/*--Outline)
							if [ -n "$dupe" ]; then
								echo $f' is a dupe!'
							else
								(echo '#'$f; echo ''; cat $f; echo ' [Source]'; echo $clipLink; echo '';  echo '========') >> $outputFile
							fi
						done
				done

			#PostsSearch
			cd /path/to/Posts
			directoryList=$(cat Directories)
			outputFile=/path/to/output/folder/$topic-$pullDate-POSTS
			outputTOC=/path/to/output/folder/$topic-$pullDate-POSTS-TOC
			echo 'Searching posts for '$topic'......'

			for d in $directoryList
				do
					cd $d
					fileList=$(ls -t)
					for f in $fileList
						do
							pulled=$(grep -i -l -H -e $topic ./$f)
							for i in $pulled
								do
									(echo '#'$f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
								done
						done
				done
			#CommsSearch
			cd /path/to/comments/archive
			fileList=$(ls -t)
			outputFile=/path/to/output/folder/$topic-$pullDate-COMMS
			outputTOC=/path/to/output/folder/$topic-$pullDate-COMMS-TOC
			echo 'Searching comments for '$topic'......'

			for f in $fileList
				do
					pulled=$(grep -i -l -H -e $topic ./$f)
			#		echo 
					for i in $pulled
						do
							(echo '#'$f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
						done
				done

			#PQSearch
			cd /path/to/config/folder
			directoryList=$(cat Directories)
			outputFile=/path/to/output/folder/$topic-$pullDate-PQS
			outputTOC=/path/to/output/folder/$topic-$pullDate-PQS-TOC
			echo 'Searching PQs for '$topic'......'

			for d in $directoryList
				do
					cd $d
					fileList=$(ls -t --hide=ISched* --hide=Posted --hide=SchedNext*)
					for f in $fileList
						do
							pulled=$(grep -i -l -H -e $topic ./$f)
							for i in $pulled
								do
									(echo '#'$f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
								done
						done
				done
	done
else
	echo 'No project specified'
fi

Usage Example:

Edit: Edit the "topicList" file so that it contains the keywords you want to search for (one per line).

Type: sh filename.sh LIES

Expected Output: This will create a text file with the name "<Keyword>-<TodaysDate>" which contains all the news clippings with text matching the specified key word(s). Other archives (Comments, Posts, Quotes) will have their own results files and have "-COMMS", "-POSTS", and "-PQS" added to the end of the filename. A file with the same naming convention (and appended with "-TOC" will also be created containing the filenames of each matching item. Each line in the TopicList file will have it's own set of results file generated.

NOTE: This will check all matching news clips against the already used sources listed in the "--Outline" files of the LIES project folder. To request a rebuild of the outlines before processing instead type: sh filename.sh LIES rebuild

You will also need to have set up the relevant "Directories" files. I keep my Quotes and my Posts in multiple folders so I add the path to each folder to the "Directories" file (one per line). The script reads this file and scans through each directory provided.

Styles

Book Stylesheet

/*original text*/
.mytext {
    display: block;
    margin: 1em 0;
    text-align: justify;
    font-family: sans-serif;
    font-size: 0.9rem;
    }
/*Blockquotes*/
.excerpt {
    font-style: italic;
    font-weight: bold;
    text-align: justify;
    margin: 0 2rem;
    font-family: sans-serif;
    font-size: 0.9rem;
    }
/*Quotes and Quote Authors*/
.pq {
    font-style: italic;
    text-align: justify;
    page-break-after: avoid;
    margin: 0 2rem;
    font-family: sans-serif;
    font-size: 0.9rem;
    }
.pqa {
   text-align: right;
   font-style: italic;
   font-family: serif;
   font-weight: normal;
   page-break-before: avoid;
   font-size: 1.1rem;
   }
/*Cut paragraph*/
.qcut {
    margin: -0.7rem 0 -0.2rem 0;
    text-align: center;
  }
/*Headings*/
/*H1*/
.calibre1 {
    font-family: "Liberation Serif", serif;
    font-size: 1.7rem;
    font-weight: bold;
    text-align: center;
    }
/*H2*/
.monoh2 {
    text-align: center;
    font-family: "Liberation Mono", monospace;
    font-size: 1.3rem
    }
/*Optional bullet formatting*/    
.bullet {
    margin: 0 2rem;
    text-align: left;
  }
.jbullet {
    margin: 0 2rem;
    text-align: justify;
  }  
/*Poetry*/
.poem {
    margin: 0 2rem;
    font-style: italic;
    text-align: justify;
    font-family: serif;
    font-size: 0.9rem; 
  }
/*Credits*/
.thanks {
    text-align: center;
    font-style: italic;
    font-family: "Liberation Serif", serif;
  }
.thanked {
    margin: 0.5rem;
    font-size: 1.4rem;
    text-align: center;
    font-style: italic;
    font-family: "Liberation Serif", serif;
  }
.thanksnote {
    text-align: justify;
    font-style: italic;
    font-family: "Liberation Serif", serif;
  }
/*Text titlepage*/
.titletext {
    padding-top: 40%;    
  }
/*Chapter headings in TOC*/
.level2 {
    font-style: italic;
    padding-left: 0.3rem;
  }
  
/*auto-added*/
.calibre {
    display: block;
    font-size: 10px;
    padding-left: 0;
    padding-right: 0;
    margin: 0 5pt
    }
.softbreak {
    display: block;
    page-break-before: avoid;
    text-align: center;
    margin: 0.5em 0 1em
    }
.whitespace {
    display: block;
    text-align: center;
    margin: 0
    }
.calibre2 {
    font-style: italic
    }
.calibre3 {
    font-size: 1.3rem;
    }

Book Page_Styles

@page {
    margin-bottom: 5pt;
    margin-top: 5pt
    }
@font-face {
  src: url(LiberationMono-Regular.ttf);
  font-family: "Liberation Mono";
  font-weight: normal;
  font-style: normal;
  font-stretch: normal;
  }
@font-face {
  src: url(LiberationSerif-Regular.ttf);
  font-family: "Liberation Serif";
  font-weight: normal;
  font-style: normal;
  font-stretch: normal;
  }
@font-face {
  src: url(LiberationSans-Regular.ttf);
  font-family: "Liberation Sans";
  font-weight: normal;
  font-style: normal;
  font-stretch: normal;
  }

Regex

Book Creation

Search

Replace

^([.][.][.])\n</p><p class="qcut">...</p><p>
<p>(?=(.*</p>\n)*<blockquote>)<p class="mytext">
<blockquote>(?=\n.*\n[-][-].*</p>\n</blockquote>)<blockquote class="pq">
[-][-](?=.*</p>\n</blockquote>)</p><p class="pqa">--
<blockquote><blockquote class="excerpt">
\[Source\]\nhttp<a href="http
(?<=http.*)</p>">[Source]</a></p>

Archived

Topic Summary (v1)

#!/bin/sh

#Change current working directory to news archive folder
cd /path/to/news/archive

#Set variables
fileList=$(grep -l -H -e $1 ./* | grep -v -e DailyNews -e Headlines-)
topic=$1
pullDate=$( date -I )
outputFile=/path/to/output/folder/$topic-$pullDate
outputTOC=/path/to/output/folder/$topic-$pullDate-TOC

#Combine found clips
for f in $fileList
	do
		(echo $f; echo ''; cat $f; echo ''; echo '========') >> $outputFile
		echo $f >> $outputTOC
	done
publicall-rights-reserved