Linux bash tiny-tips

Killin’ the process via its id in Linux:
#ps -aux | grep IVI_ConnectionManager | awk ‘NR==1{print $2}’ | xargs kill

 

Launch two different shell in one script:

#/bin/bash

command_1 && command_2

(Notice the & sign at the end of each line. This will cause the shell to fork that process into the background and continue execution. Note how it’s different from &&, which is basically an and sign, command_1 && command_2 will executecommand_1 and if it exits with success, only then run command_2, while command_1 & command_2 will start the second right after the first.)

 

Get input repeatedly from the shell:

Value 127 (non-zero) indicates command cyberciti failed to execute. You can use exit status in shell scripting too. You can store result of exit status in variable. Consider following shell script:

#!/bin/bash
echo -n "Enter user name : "
read USR
cut -d: -f1 /etc/passwd | grep "$USR" > /dev/null
OUT=$?
if [ $OUT -eq 0 ];then
   echo "User account found!"
else
   echo "User account does not exists in /etc/passwd file!"
fi

Save and execute the script as follows:
$ chmod +x script.sh
$ ./script.sh

Output:

Enter user name : jradmin
User account does not exists in /etc/passwd file

Try it one more time:
$ ./script.sh
Output:

Enter user name : vivek
User account found

 

you can filter out messages to stderr. I prefer to redirect them to stdout like this.

find / -name art  2>&1 | grep -v “Permission denied”

Explanation:

In short, all regular output goes to standard output (stdout). All error messages to standard error (stderr).

grep usually finds/prints the specified string, the -v inverts this, so it finds/prints every string that doesn’t contain “Permission denied”. All of your output from the find command, including error messages usually sent to stderr (file descriptor 2) go now to stdout(file descriptor 1) and then get filtered by the grep command.

This assumes you are using the bash/sh shell.

Under tcsh/csh you would use

find / -name art |& grep ….

 

Shell:

jiafei427@CKUBU:~/workspace/kernel/modules/test$ cat infinity_touch.sh
i=0
while [ 1 ]
do
echo “wocao”
sleep 0.01
i=i+1
done

That will repeat print wocao with sleeping 0.01 second

 

create folder with certain format of date

mkdir "$(date +"%d-%m-%Y")"
cd "$(date +"%d-%m-%Y")"

In the extreme case a day passes between the first and the second statement, that won’t work. Change it to:

d="$(date +"%d-%m-%Y")"
mkdir "$d"
cd "$d"

Explanation: The $(...) returns the output from the subcommands as a string, which we store in the variable d.

 

Getting shell variables into awk may be done in several ways. Some are better than others.


This is the best way to do it. It uses the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar)

variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two

This should be compatible with most awk and variable is available in the BEGIN block as well:

 

 

Read full link from symbolic link file:

readlink -f `which command`

If command is in your $PATH variable , otherwise you need to specify the path you know.

You can use awk with a system call readlink to get the equivalent of an ls output with full symlink paths. For example:

ls | awk '{printf("%s ->", $1); system("readlink -f " $1)}'

Will display e.g.

thin_repair ->/home/user/workspace/boot/usr/bin/pdata_tools
thin_restore ->/home/user/workspace/boot/usr/bin/pdata_tools
thin_rmap ->/home/user/workspace/boot/usr/bin/pdata_tools
thin_trim ->/home/user/workspace/boot/usr/bin/pdata_tools
touch ->/home/user/workspace/boot/usr/bin/busybox
true ->/home/user/workspace/boot/usr/bin/busybox

 

awk  system output to certain variable:

To run a system command in awk you can either use system() or cmd | getline.

I prefer cmd | getline because it allows you to catch the value into a variable:

$ awk 'BEGIN {"date" |  getline mydate; close("date"); print "returns", mydate}'
returns Thu Jul 28 10:16:55 CEST 2016

More generally, you can set the command into a variable:

awk 'BEGIN {
       cmd = "date -j -f %s"
       cmd | getline mydate
       close(cmd)
     }'

Note it is important to use close() to prevent getting a “makes too many open files” error if you have multiple results (thanks mateuscb for pointing this out in comments).

 

Note: Coprocess is GNU awk specific. Anyway another alternative is using getline

cmd = "strip "$1
while ( ( cmd | getline result ) > 0 ) {
  print  result
} 
close(cmd)
or sth. like this:
awk 'BEGIN{"date"|getline d; print "Current date is:" , d }'

 

list all symbolic links in a directory:

Parsing ls is a Bad Idea®, prefer a simple find in that case:

find . -type l -ls

To only process the current directory:

find . -maxdepth 1 -type l -ls

 

 

How to find and list all the symbolic links created for a particular file?

Here is an example:

find -L /dir/to/start -xtype l -samefile ~/Pictures

or, maybe better:

find -L /dir/to/start -xtype l -samefile ~/Pictures 2>/dev/null

to get rid of some errors like Permission deniedToo many levels of symbolic links, or File system loop detected which find throws them when doesn’t have the right permissions or other situations.

  • -L – Follow symbolic links.
  • -xtype l – File is symbolic link
  • -samefile name – File refers to the same inode as name. When -L is in effect, this can include symbolic links.

 

 

 

1. Replacing all occurrences of one string with another in all files in the current directory:

These are for cases where you know that the directory contains only regular files and that you want to process all non-hidden files. If that is not the case, use the approaches in 2.

All sed solutions in this answer assume GNU sed. If using FreeBSD or OS/X, replace -i with -i ''. Also note that the use of the -i switch with any version of sed has certain filesystem security implications and is inadvisable in any script which you plan to distribute in any way.

  • Non recursive, files in this directory only:
    sed -i -- 's/foo/bar/g' *
    perl -i -pe 's/foo/bar/g' ./* 

    (the perl one will fail for file names ending in | or space)).

  • Recursive, regular files (including hidden ones) in this and all subdirectories
    find . -type f -exec sed -i 's/foo/bar/g' {} +

    If you are using zsh:

    sed -i -- 's/foo/bar/g' **/*(D.)

    (may fail if the list is too big, see zargs to work around).

    Bash can’t check directly for regular files, a loop is needed (braces avoid setting the options globally):

    ( shopt -s globstar dotglob;
        for file in **; do
            if [[ -f $file ]] && [[ -w $file ]]; then
                sed -i -- 's/foo/bar/g' "$file"
            fi
        done
    )

    The files are selected when they are actual files (-f) and they are writable (-w).

2. Replace only if the file name matches another string / has a specific extension / is of a certain type etc:

  • Non-recursive, files in this directory only:
    sed -i -- 's/foo/bar/g' *baz*    ## all files whose name contains baz
    sed -i -- 's/foo/bar/g' *.baz    ## files ending in .baz
  • Recursive, regular files in this and all subdirectories
    find . -type f -name "*baz*" -exec sed -i 's/foo/bar/g' {} +

    If you are using bash (braces avoid setting the options globally):

    ( shopt -s globstar dotglob
        sed -i -- 's/foo/bar/g' **baz*
        sed -i -- 's/foo/bar/g' **.baz
    )

    If you are using zsh:

    sed -i -- 's/foo/bar/g' **/*baz*(D.)
    sed -i -- 's/foo/bar/g' **/*.baz(D.)

    The -- serves to tell sed that no more flags will be given in the command line. This is useful to protect against file names starting with -.

  • If a file is of a certain type, for example, executable (see man find for more options):
    find . -type f -executable -exec sed -i 's/foo/bar/g' {} +

    zsh:

    sed -i -- 's/foo/bar/g' **/*(D*)

3. Replace only if the string is found in a certain context

  • Replace foo with bar only if there is a baz later on the same line:
    sed -i 's/foo\(.*baz\)/bar\1/' file

    In sed, using \( \) saves whatever is in the parentheses and you can then access it with \1. There are many variations of this theme, to learn more about such regular expressions, see here.

  • Replace foo with bar only if foo is found on the 3d column (field) of the input file (assuming whitespace-separated fields):
    gawk -i inplace '{gsub(/foo/,"baz",$3); print}' file

    (needs gawk 4.1.0 or newer).

  • For a different field just use $N where N is the number of the field of interest. For a different field separator (: in this example) use:
    gawk -i inplace -F':' '{gsub(/foo/,"baz",$3);print}' file

    Another solution using perl:

    perl -i -ane '$F[2]=~s/foo/baz/g; $" = " "; print "@F\n"' foo 

    NOTE: both the awk and perl solutions will affect spacing in the file (remove the leading and trailing blanks, and convert sequences of blanks to one space character in those lines that match). For a different field, use $F[N-1] where N is the field number you want and for a different field separator use (the $"=":" sets the output field separator to :):

    perl -i -F':' -ane '$F[2]=~s/foo/baz/g; $"=":";print "@F"' foo 
  • Replace foo with bar only on the 4th line:
    sed -i '4s/foo/bar/g' file
    gawk -i inplace 'NR==4{gsub(/foo/,"baz")};1' file
    perl -i -pe 's/foo/bar/g if $.==4' file

4. Multiple replace operations: replace with different strings

  • You can combine sed commands:
    sed -i 's/foo/bar/g; s/baz/zab/g; s/Alice/Joan/g' file

    Be aware that order matters (sed 's/foo/bar/g; s/bar/baz/g' will substitute foo with baz).

  • or Perl commands
    perl -i -pe 's/foo/bar/g; s/baz/zab/g; s/Alice/Joan/g' file
  • If you have a large number of patterns, it is easier to save your patterns and their replacements in a sed script file:
    #! /usr/bin/sed -f
    s/foo/bar/g
    s/baz/zab/g
  • Or, if you have too many pattern pairs for the above to be feasible, you can read pattern pairs from a file (two space separated patterns, $pattern and $replacement, per line):
    while read -r pattern replacement; do   
        sed -i "s/$pattern/$replacement/" file
    done < patterns.txt
  • That will be quite slow for long lists of patterns and large data files so you might want to read the patterns and create a sed script from them instead. The following assumes a <space>delimiter separates a list of MATCH<space>REPLACE pairs occurring one-per-line in the file patterns.txt :
    sed 's| *\([^ ]*\) *\([^ ]*\).*|s/\1/\2/g|' <patterns.txt |
    sed -f- ./editfile >outfile

    The above format is largely arbitrary and, for example, doesn’t allow for a <space> in either of MATCH or REPLACE. The method is very general though: basically, if you can create an output stream which looks like a sed script, then you can source that stream as a sed script by specifying sed‘s script file as -stdin.

  • You can combine and concatenate multiple scripts in similar fashion:
    SOME_PIPELINE |
    sed -e'#some expression script'  \
        -f./script_file -f-          \
        -e'#more inline expressions' \
    ./actual_edit_file >./outfile

    A POSIX sed will concatenate all scripts into one in the order they appear on the command-line. None of these need end in a \newline.

  • grep can work the same way:
    sed -e'#generate a pattern list' <in |
    grep -f- ./grepped_file
  • When working with fixed-strings as patterns, it is good practice to escape regular expression metacharacters. You can do this rather easily:
    sed 's/[]$&^*\./[]/\\&/g
         s| *\([^ ]*\) *\([^ ]*\).*|s/\1/\2/g|
    ' <patterns.txt |
    sed -f- ./editfile >outfile

5. Multiple replace operations: replace multiple patterns with the same string

  • Replace any of foobar or baz with foobar
    sed -Ei 's/foo|bar|baz/foobar/g' file
  • or
    perl -i -pe 's/foo|bar|baz/foobar/g' file

 

 

skip line containing certain expression


#cat file_name | awk '$0 ~ /Makefile$/'

that will print out the line without line ending with Makefile

 

To remove the line and print the output to standard out:

sed '/pattern to match/d' ./infile

To directly modify the file:

sed -i '/pattern to match/d' ./infile

To directly modify the file (and create a backup):

sed -i.bak '/pattern to match/d' ./infile

For Mac OS X users:

sed -i '' '/pattern/d' ./infile

 

Print Text files only without binary files:

I know this is an old thread, but I stumbled across it and thought I’d share my method which I have found to be a very fast way to use find to find only non-binary files:

find . -type f -exec grep -Iq . {} \; -and -print

The -I option to grep tells it to immediately ignore binary files and the . option along with the -q will make it immediately match text files so it goes very fast. You can change the -print to a -print0 for piping into an xargs -0 or something if you are concerned about spaces (thanks for the tip, @lucas.werkmeister!)

Also the first dot is only necessary for certain BSD versions of find such as on OS X, but it doesn’t hurt anything just having it there all the time if you want to put this in an alias or something.

 

Replace certain string for all files in a directory:

# find ./ -type f -exec sed -i ‘s/string1/string2/g’ {} \;

 

…TBD

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s