nixCraft

jaysunn · #1 (**permalink**) 01-10-2010, 06:33 PM

Hello Friends,

Here I will show you a technique using SED to replace a text string with a new string in multiple files. The files that I will be altering are shell scripts that perform for loops to perform different tasks. When we add a new server to the cluster, we would have to manually open each shell script and the new server. This script will automate the process saving the admin precious time.

The shell script resides in /usr/local/bin on our staging server. There are 50 of them.

Code:

[root@staging1 bin]# ls
abc.sh   abcd.sh abcde.sh abcdef.sh abcdefg.sh abcdefgh.sh abcdefghij.sh

Lets look at the contents of a for loop script in this directory.

PHP Code:


			
[root@staging1 bin]# cat rsync_radiox_cluster.sh
#!/bin/bash

for server in "server1" "server2.nyc" "server3.nyc" ;        
do   

/usr/bin/rsync  -e ssh -rlvtzgoD /dir1/dir2/dir3/ root@$server:/dir1/dir2/dir3/
done

Now I will show you the sed script to add the new server to all of the shell scripts in /usr/local/bin.

PHP Code:


			
[root@staging1 bin]# cat sed_string_replace.sh 
#!/bin/bash
# Jason R. Ralph
# November 20th 2009

for scripts in $(find /usr/local/bin/ -name "*sh" | awk -F "/" '{print $NF}') ; do

cat $scripts | sed 's/"server3.nyc"/"server3.nyc" "server4.nyc"/g' > /tmp/$scripts.tmp

cp -f /tmp/$scripts.tmp  /usr/local/bin/$scripts

rm -f /tmp

After you run the script you will see that all the shell scripts now have server4.nyc in them.

PHP Code:


			
[root@staging1 bin]# cat rsync_radiox_cluster.sh 
#!/bin/bash 

for server in "server1" "server2.nyc" "server3.nyc" "server4.nyc" ;         
do    

/usr/bin/rsync  -e ssh -rlvtzgoD /dir1/dir2/dir3/ root@$server:/dir1/dir2/dir3/ 
done

You can modify the search criteria as well as the replace portion for your needs. You will have to alter this piece of the script. The search string is red. And the replace string is blue.

Code:

sed 's/"server3.nyc"/"server3.nyc" "server4.nyc"/g'

Please let me know if you have questions or concerns for this lesson.

HTH,

Jaysunn

ghostdog74 · #2 (**permalink**) 01-10-2010, 08:06 PM

1) your for loop with the find + awk command will break on files with spaces.
2) lose the cat. pass the file into sed directly

here's a version, using just shell and no external tool and assuming your *.sh files are not all over the place but just in /usr/local/bin

Code:

#!/bin/bash
shopt -s nullglob
for file in /usr/local/bin/*.sh
do
    while read -r line
    do
        case "$line" in
            *"for server in \"server1"* )
                line=${line%;}
                line="$line \"server4.nyc\" ;";;
        esac
        echo $line >> temp
    done < "$file"
    mv temp "$file"
done

or if you still want to use sed

Code:

sed -i.bak 's/server3.nyc ; /server3.nyc server4.nyc;/' *.sh

jaysunn · #3 (**permalink**) 01-10-2010, 08:27 PM

Hello,

I have been looking at the man page for shopt. However can you briefly explain what you are achieving with this line:

Code:

shopt -s nullglob

Thanks,

Jaysunn

nixcraft · #4 (**permalink**) 01-10-2010, 08:55 PM

Quote:

Originally Posted by jaysunn

Hello,

I have been looking at the man page for shopt. However can you briefly explain what you are achieving this this line:

Code:

shopt -s nullglob

Thanks,

Jaysunn

If there are no *.sh file in /usr/bin an error message will be displayed on screen and the script is not executed. It is kind of failsafe stuff. See

Use BASH nullglob To Verify *.c Files Exists or Not In a Directory

A simple example:

Code:

cd /tmp
touch x{1,2}.fb
var="*.fb"
echo $var
rm *.fb
var="*.fb"
# if there are no *.fb var will store *.fb itself
echo $var
shopt -s nullglob
var="*.fb"
echo $var

jaysunn · #5 (**permalink**) 01-10-2010, 09:28 PM

Hello Ghostdog,

I have attempted to use your script to add a new server to my for loop. However I am not seeing the scripts update in /usr/local/bin. Here is the example that I used.

Directory Contents:

Code:

[root@radio5 shelltest]# ls
script1.sh script2.sh script3.sh script4.sh script5.sh script6.sh

Each script contains something similar to this:

Code:

[root@staging1 shelltest]# cat remote_command.sh 
#!/bin/bash
#display usage
if [ -z "$1" ]; then
    echo -e usage: "Command \n" 
    exit
fi
  
for server in "server1.nyc" "server2.nyc" "server3.nyc"  ;        

do



echo -e "$server \n"
  ssh -l root $server "$1"
done
[root@staging1 shelltest]#

My goal is to add server4.nyc to the for loop in portion. I ran this:

Code:

root@staging1 shelltest]# cat tester.sh 
#!/bin/bash
shopt -s nullglob
for file in /usr/local/bin/*.sh
do
    while read -r line
    do
        case "$line" in
            *"for server in \"server3.nyc"* )
                line=${line%;}
                line="$line \"server3.nyc" "server4.nyc\" ;";;
        esac
        echo $line >> temp
    done < "$file"
    mv temp "$file"
done
[root@staging1 shelltest]#

My expected results are:

Code:

[root@staging1 shelltest]# cat remote_command.sh 
#!/bin/bash
#display usage
if [ -z "$1" ]; then
    echo -e usage: "Command \n" 
    exit
fi
  
for server in "server1.nyc" "server2.nyc" "server3.nyc" "server4.nyc" ;        

do

echo -e "$server \n"
  ssh -l root $server "$1"
done
[root@staging1 shelltest]#

The command completed successfully. However when checking the shell scripts, I did not see the new server. Am I running this incorrectly?

Please Advise.

Jaysunn

jaysunn · #6 (**permalink**) 01-10-2010, 09:46 PM

@ghostdog,

Your script worked great. It was my fault. I ran it from a test directory. However I was not looking at /usr/local/bin to verify since I tested in a different location.

One thing I noticed was my executable permissions were different on the changed scripts.

Sorry for the confusion. Any Idea why this could of occurred?

Thanks for all your help.

Jaysunn

ghostdog74 · #7 (**permalink**) 01-11-2010, 06:26 AM

i don't understand. please paste a copy of the permissions before and after here.

nixcraft · #8 (**permalink**) 01-11-2010, 12:39 PM

mv may change the file permission. Are you deleting or moving exiting file before running mv? If so umask will affect it.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
To find and replace in text file	nashtech	Shell scripting	5	01-31-2010 06:32 AM
[Solved] Read and replace text in html tags between 2 files	aixjadoo	Shell scripting	2	12-13-2009 05:20 PM
How to use sed to replace text between html tags	raj	Shell scripting	2	11-20-2009 07:24 PM
perl replace text	kasimani	CentOS / RHEL / Fedora	3	11-10-2009 07:56 PM
How to list and replace file of same name in multiple paths	shchandu	Shell scripting	1	08-27-2009 07:32 PM

nixCraft

Replace Text String Multiple Files With SED

Replace Text String Multiple Files With SED

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads

The Following User Says Thank You to ghostdog74 For This Useful Post:
jaysunn (01-10-2010)