SED Search Engine and Regular Expression

Top Regex

Strip HTML Tag with Sed - First Method ( remove head/tail )

wget -q -O - checkip.dyndns.org
<html><head><title>Current IP Check</title></head><body>Current IP Address: 87.2.146.126</body></html>
wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //'
87.2.146.126</body></html>
wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//'
87.2.146.126

Strip HTML Tag with Sed - Second Method ( remove all string between <> tags )

wget -q -O - checkip.dyndns.org
<html><head><title>Current IP Check</title></head><body>Current IP Address: 87.2.146.126</body></html>
wget -q -O - checkip.dyndns.org|sed -e 's/<[^>]*>//g' 
Current IP CheckCurrent IP Address: 87.2.146.126

Strip HTML Tag with Sed - Third Method ( extract between tags )

echo "gfsagfgssaaaabbbbccccasdghfsd" | sed -e 's/.*aaaa\(.*\)cccc.*/\1/'
bbbb

echo "gfsagfgss<aaaa>bbbb</aaaa>asdghfsd" | sed -e 's/.*<aaaa>\(.*\)<\/aaaa>.*/\1/'
bbbb

wget -q -O - checkip.dyndns.org | sed -e 's/.*Current IP Address: \(.*\)<\/body>.*/\1/'
87.2.146.126

Extract Url Function

echo 'http://www.example.com/hello?aaa=bbb&ddd=ffff' | sed -e 's/.*\/\([^?]*\).*/\1/'
hello

Extract Url Parameters

echo 'http://www.example.com/hello?aaa=bbb&ddd=ffff' | sed  's/.*?//'
aaa=bbb&ddd=ffff
echo 'http://www.example.com/hello?aaa=bbb&ddd=ffff' | sed  's/.*?/?/'
?aaa=bbb&ddd=ffff

HTML Entities

cat cds_response_20140101164508.xml | sed -e 's/&lt;/</g' -e 's/&quot;/"/g' -e 's/&gt;/>/g'

http://w3schools.com/html/html_entities.asp

http://w3schools.com/tags/ref_entities.asp

File extention regex

echo abc.txt.doc.avi | sed 's/^.*\.//'
avi
echo abc.txt.doc | sed 's/^.*\.//'
doc
echo abc.txt | sed 's/^.*\.//'
txt
echo abc | sed 's/^.*\.//'
abc

Extract HTML Links

wget -q -O - http://delicious.com/rkpisanu  | sed 's/^.*<a href="//' | sed 's/".*$//' | grep ^http
https://secure.delicious.com/register
https://secure.delicious.com/login?jump=http%3A%2F%2Fdelicious.com%2Frkpisanu
http://feeds.delicious.com/v2/rss/rkpisanu?count=15
http://blog.delicious.com/
http://info.yahoo.com/privacy/us/delicious/
http://info.yahoo.com/copyright/details.html
http://support.delicious.com/forum/

Strip Space

echo "      aaaaaa"
    aaaaaa
echo "      aaaaaa" | sed -e 's/^[[:space:]]*//'
aaaaaa

Get string within ""

echo aggfsfd\"aaaaa\"ghgmdhf
aggfsfd"aaaaa"ghgmdhf
echo aggfsfd\"aaaaa\"ghgmdhf | awk -v RS=\" '!(NR%2)'
aaaaa

Insert a newline after >

echo "<one><two><three>"
<one><two><three>
echo "<one><two><three>" | sed "s/>/>\\`echo -e '\n\r'`/g"
<one>
<two>
<three>

Insert a newline after , ( Useful for youtube scraper )

echo "aaaa,bbb,ccc,dddd"
aaaa,bbb,ccc,dddd

echo "aaaa,bbb,ccc,dddd"  | sed "s/,/\n/g"
aaaa
bbb
ccc
dddd

Remove a newline ( Useful for youtube scraper )

echo -e "a\nb\nc"
a
b
c
echo -e "a\nb\nc" | tr -d '\n'
abc

Substitute a newline with space ( Useful for sql query format )

cat file.txt
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
 ddddd  pergajr
cat file.txt | awk '{ print $1"," }'  | tr '\n' ' ' ; echo " "
aaaa, bbb, cccc, ddddd,  

Substitute a newline with space and add quote in the same line ( Useful for sql query format )

cat file.txt                                                  
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
 ddddd  pergajr
cat file.txt | awk '{ print "\047"$1"\047""," }' | tr '\n' ' ' ; echo " "
'aaaa', 'bbb', 'cccc', 'ddddd',  

Substitute a newline with space and add quote not in the same line ( Useful for sql query format )

cat file.txt                                                  
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
 ddddd  pergajr
cat file.txt | awk '{ print "\047"$1"\047""," }'
'aaaa', 
'bbb', 
'cccc', 
'ddddd',  

Search a line with a tag and comment it

sed 's%TAG_HERE%#&%g' file
cat file.txt                                                             
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
 ddddd  pergajr
cat file.txt | sed 's%cccc%#&%g'
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 #cccc  fpgfsisdf
 ddddd  pergajr

Search a line with a tag and delete it

cat file.txt                                                             
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
 ddddd  pergajr
cat file.txt | sed "/cccc/d"
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 ddddd  pergajr

Extract part of text file with first and last tag

cat file.txt
aaaaa
bbbbb
ccccc
ddddd
eeeee
fffff
ggggg
awk '/ccccc/,/eeeee/' file.txt
ccccc
ddddd
eeeee

Add lines after a tag

cat file.txt           
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
 ddddd  pergajr
cat add.cmd 
/cccc/a\
pippo\
pluto\
paperino
sed -f add.cmd file.txt
 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
 cccc   fpgfsisdf
pippo
pluto
paperino
 ddddd  pergajr

Single line:

sed '/bbb/a \
eeeeeeee' file.txt

 aaaa   lfkgnsdffgn
 bbb    pfdmnsbmdsf
eeeeeeee
 cccc   fpgfsisdf
 ddddd  pergajr

Escaping...

Top Searching with find

Find a file without error permission

find / -iname ls 2>/dev/null

Search for files changed between x and y minute before

Search for files changed between 0 and 1 minute before,
find . -cmin 1
between 1 and 2 minute before,
find . -cmin 2

Search for files or copy files with grep

find . -name "*" -exec grep sql \{\} \;
find ./ -name "*.ksh" -exec grep sql {} \; -ls   
find .  -name '*' -exec grep -l "sql" {} \; -exec cp {} /tmp/paolo \;
find .  -name '*' -exec grep -l "sql" {} \; -exec ls \;
find /tmp/paolo -name "*" -exec cp {} . \;

Search string nfs in only text files with grep

find ./ -name "*" -type f -exec grep -I "nfs" {} \; -ls
for file in $(find . -name "*" -type f -print | xargs file | grep ASCII | cut -d: -f1)
do
echo $file
grep sdcard $file
done

Create command search and put in /usr/bin:

vi search
find ./ -name "*" -type f -exec grep -I $1 {} \; -ls

chmod +x search
sudo mv ./search /usr/bin

With grep…

grep -r "func_upload" * 

Top Shell Tips

Logging with timestamp

TEMPDIR="tempdir"
mkdir -p $HOME/${TEMPDIR}
LOGFILE="$HOME/${TEMPDIR}/wdtv_install.log"
echo `date +"%Y%m%d%H%M%S"`"|INFO|-->Trace: finalize" >>${LOGFILE} 2>&1

Command-line interactive programs in UNIX shell-scripts

echo p | svn co --username $1 --password $2 $3 $4 >>${LOGFILE} 2>&1

Ext Selection

file=$1
ext=`echo ${file##*.} | tr '[:upper:]'  '[:lower:]'`
case $ext in
  zip ) # Zip Files 
      echo `date +"%Y%m%d%H%M%S"`"|INFO|-->Trace:unzip_file:Unzip file $1 has an extension zip" >>${LOGFILE} 2>&1
      unzip -u $1 >>${LOGFILE} 2>&1
      ;;
  tgz ) # Tgz Files 
      echo `date +"%Y%m%d%H%M%S"`"|INFO|-->Trace:unzip_file:Unzip file $1 has an extension tgz" >>${LOGFILE} 2>&1
      tar xvfz $1 >>${LOGFILE} 2>&1
      ;;
esac

Top Shell Links

Bash Hackers Wiki

unix/regex.txt · Last modified: 2014/07/18 16:31 by rkpisanu
www.chimeric.de Creative Commons License Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0





Mail