Page 1 of 2 1 2 LastLast
Results 1 to 10 of 12

Thread: Shell Script to monitor folder and upload found files via FTP

  1. #1
    Junior Member
    Join Date
    Jul 2010
    Posts
    5
    Thanks
    0
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Default Shell Script to monitor folder and upload found files via FTP

    Hi everyone!

    I'm in a need of a shell script that search for all files in a folder, move all those files to a temp folder, and upload those files via FTP. When the file transfer via FTP completes successfully, the file is moved to a completed folder. In case any of those files fails, the file will be moved to a failed folder.

    I have very little knowledge on Shell Scripting, but I'll try to explain as clear as possible what I need using a example flow below.

    Code:
    #!/bin/sh
    
    # Configure variables at the begining of the script
    MONITOR_DIR=/path/to/monitor/files/
    TMP_DIR=/path/to/monitor/files/temp/
    SUCCESS_DIR=/path/to/monitor/files/success/
    FAILED_DIR=/path/to/monitor/files/failed/
    PROCCESS_ID_DIR={execution timestamp string}
    FILE_EXT=flv
    
    # FTP variables
    FTP_HOST=ftp.myserver.com
    FTP_USER=mysername
    FTP_PASS=mypassword
    FTP_REMOTE_DIR=/

    The script will work with these steps:

    1. For each file with extension $FILE_EXT in $MONITOR_DIR
    2. echo "found file: " and the name(s) of file(s) found
    3. Move all found files to $TMP_DIR/$PROCCESS_ID_DIR (folder $PROCCESS_ID_DIR has to be created)
    4. For each file in $TMP_DIR/$PROCCESS_ID_DIR
    5. Upload via FTP each file individually using the FTP variables
    6. If FTP transfer completes, move that file to $SUCCESS_DIR, else move it to $FAILED_DIR
    7. echo "file (file_name) transfered" or "file (file_name) failed" depending on what happened above
    8. When all the files in $TMP_DIR/$PROCCESS_ID_DIR are processed, delete $PROCCESS_ID_DIR folder in $TMP_DIR
    9. quit script


    That's basically it. I need this to monitor/backup/copy-to-CDN files uploaded to my site using a cron file.

    Any help will be greatly appreciated...

    Thanks!


  2. The Following User Says Thank You to pulsorock For This Useful Post:

    linux12 (23rd July 2010)

  3. #2
    Member
    Join Date
    Jul 2010
    Posts
    31
    Thanks
    4
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Default

    added me
    # Configure variables at the begining of the script
    MONITOR_DIR=/home/
    TMP_DIR=/tmp
    SUCCESS_DIR=/var/log/ftpmoniter/ftp-SUCCESS.log
    FAILED_DIR=/var/log/ftpmoniter/ftp-FAILED.log
    ipuserupload=/var/log/ftpmoniter/ipuserupload.log
    PROCCESS_ID_DIR={execution timestamp string}
    FILE_EXT=php,cgi,pl,
    privintnamefileupload=c99,r57,phpshell,cgi-telnet
    and
    drop "/sbin/iptables -A INPUT -s %s -j DROP"


    Why not be on all accounts on the FTP server is not determined by one account
    Last edited by linux12; 23rd July 2010 at 09:19 PM.

  4. #3
    Member
    Join Date
    Jul 2010
    Posts
    31
    Thanks
    4
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Default

    The idea is really beautiful
    I'm not an expert programmer Shell I'm just a beginning programmer
    Provided you some tips to be better
    Waiting for the experts to assist in the writing bash
    And thank you for this great idea

  5. #4
    Never say die nixcraft's Avatar
    Join Date
    Jan 2005
    Location
    BIOS
    Posts
    4,514
    Thanks
    17
    Thanked 808 Times in 511 Posts
    Rep Power
    10

    Default

    Don't make it complicated it is pretty easy:
    Code:
    NOW="$(date)"
    for f in /path/to/*.flv 
    do
        lftp -e "put $f -O $FTP_REMOTE_DIR" -u "${FTP_USER},${FTP_PASS}" $FTP_HOST
        if [ $? -ne 0 ]
        then
            echo "$HOSTNAME $NOW Upload failed for $f, do something" >>$SUCCESS_DIR
            cp $f 
        else
            echo "$HOSTNAME $NOW: Upload done for $f..do something here" >>$FAILED_DIR
        fi
    done
    Replace /path/to/*.flv with actual path.
    Last edited by nixcraft; 23rd July 2010 at 09:54 PM.
    All [Solved] threads are closed by mods / admin to avoid spam issues. See Howto mark a thread as [Solved]


  6. The Following User Says Thank You to nixcraft For This Useful Post:

    linux12 (23rd July 2010)

  7. #5
    Member
    Join Date
    Jul 2010
    Posts
    31
    Thanks
    4
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Default

    A little help from me

    Code:
    #!/bin/sh
    
    MONITOR_DIR=/home/
    TMP_DIR=/tmp
    SUCCESS_DIR=/var/log/ftpmoniter/ftp-SUCCESS.log
    FAILED_DIR=/var/log/ftpmoniter/ftp-FAILED.log
    ip_user_upload=/var/log/ftpmoniter/ipuserupload.log
    deny_name_file_upload=c99,r57,phpshell,cgi-telnet
    PROCCESS_ID_DIR={execution timestamp string}
    FILE_EXT_MONITOR=php,cgi,pl
    iptable_drop=/sbin/iptables -A INPUT -s %s -j DROP
    
    touch /var/log/ftpmoniter/ftp-SUCCESS.log
    chown root.root /var/log/ftpmoniter/ftp-SUCCESS.log
    touch /var/log/ftpmoniter/ftp-SUCCESS.log
    chown root.root /var/log/ftpmoniter/ftp-SUCCESS.log
    touch /var/log/ftpmoniter/ipuserupload.log
    chown root.root /var/log/ftpmoniter/ipuserupload.log
    
    _deny_name_file_upload="c99 r57 phpshell cgi-telnet
    for d in $_deny_name_file_upload
    do
        echo 
    done

  8. #6
    Junior Member
    Join Date
    Jul 2010
    Posts
    5
    Thanks
    0
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Default

    Quote Originally Posted by nixcraft View Post
    Don't make it complicated it is pretty easy:
    Code:
    NOW="$(date)"
    for f in /path/to/*.flv 
    do
        lftp -e "put $f -O $FTP_REMOTE_DIR" -u "${FTP_USER},${FTP_PASS}" $FTP_HOST
        if [ $? -ne 0 ]
        then
            echo "$HOSTNAME $NOW Upload failed for $f, do something" >>$SUCCESS_DIR
            cp $f 
        else
            echo "$HOSTNAME $NOW: Upload done for $f..do something here" >>$FAILED_DIR
        fi
    done
    Replace /path/to/*.flv with actual path.
    Thanks nixcraft,

    What do you mean when saying "Don't make it complicated"? The above code does everything? (sorry there is some code there that I'm not sure what it does). In the code above, will the files be moved when success or failed?

    On the server where this will be running, there is no lftp installed (and I have no access to install it), what would be the equivalent using the regular ftp commands?

    Thanks again!

  9. #7
    Is that all you got? rockdalinux's Avatar
    Join Date
    May 2005
    Location
    Planet Vegeta
    Posts
    987
    Thanks
    27
    Thanked 71 Times in 62 Posts
    Rep Power
    20

    Default

    Quote Originally Posted by pulsorock View Post
    In the code above, will the files be moved when success or failed?
    Noop, you have to add your command after the echo statement between if...else...fi:
    Code:
    cp "$f" /path/to/failed
    OR
    Code:
    cp "$f" /path/to/success
    Rocky Jr.
    What's wrong? I hope I am not making you uncomfortable...

    Never send a boy to do a mans job.

  10. #8
    Junior Member
    Join Date
    Jul 2010
    Posts
    5
    Thanks
    0
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Default

    Since I don't have lftp installed I tried using regular FTP and here is the code I have so far.

    Code:
    #!/bin/sh
    
    # Configure variables at the begining of the script
    MONITOR_DIR=/var/www/web12/web/php-site-monitor/files
    TMP_DIR=/var/www/web12/web/php-site-monitor/files/tmp
    SUCCESS_DIR=/var/www/web12/web/php-site-monitor/files/success
    FAILED_DIR=/var/www/web12/web/php-site-monitor/files/failed
    FILE_EXT=mp4
    
    # FTP variables
    FTP_HOST=upload-ftp.simplecdn.com
    FTP_USER=myuser
    FTP_PASS=mypass
    FTP_REMOTE_DIR=/
    
    for f in $MONITOR_DIR/*.$FILE_EXT; do
    	echo "Uploading: " $f
    	ftp -inv $FTP_HOST <<EOF
    	user $FTP_USER $FTP_PASS
    	binary
    	cd $FTP_REMOTE_DIR
    	put $f
    	bye
    	EOF
    	#mv $f $SUCCESS_DIR
    done
    I set the FTP command to Verbose to see the whole output from the server. The output I'm getting is:

    Code:
    # ./shell_php.sh
    Uploading  /var/www/web12/web/php-site-monitor/files/test_file1.mp4
    Connected to upload-ftp.simplecdn.com.
    220 Xlight FTP Server 3.5 ready...
    431 FTP server doesn't support TLS/SSL encryption
    431 FTP server doesn't support TLS/SSL encryption
    KERBEROS_V4 rejected as an authentication type
    Remote system type is UNIX.
    Using binary mode to transfer files.
    331 Password required for myuser
    230 Login OK
    200 Type set to I.
    250 Directory successfully changed
    local: /var/www/web12/web/php-site-monitor/files/test_file1.mp4 remote: /var/www/web12/web/php-site-monitor/files/test_file1.mp4
    227 Entering Passive Mode (204,10,171,131,196,208)
    451 Can't create file "test_file1.mp4".
    221 Good-Bye
    Uploading  /var/www/web12/web/php-site-monitor/files/test_file2.mp4
    Connected to upload-ftp.simplecdn.com.
    220 Xlight FTP Server 3.5 ready...
    431 FTP server doesn't support TLS/SSL encryption
    431 FTP server doesn't support TLS/SSL encryption
    KERBEROS_V4 rejected as an authentication type
    Remote system type is UNIX.
    Using binary mode to transfer files.
    331 Password required for myuser
    230 Login OK
    200 Type set to I.
    250 Directory successfully changed
    local: /var/www/web12/web/php-site-monitor/files/test_file2.mp4 remote: /var/www/web12/web/php-site-monitor/files/test_file2.mp4
    227 Entering Passive Mode (204,10,171,131,195,230)
    451 Can't create file "test_file2.mp4".
    221 Good-Bye
    Uploading  /var/www/web12/web/php-site-monitor/files/test_file3.mp4
    Connected to upload-ftp.simplecdn.com.
    220 Xlight FTP Server 3.5 ready...
    431 FTP server doesn't support TLS/SSL encryption
    431 FTP server doesn't support TLS/SSL encryption
    KERBEROS_V4 rejected as an authentication type
    Remote system type is UNIX.
    Using binary mode to transfer files.
    331 Password required for myuser
    230 Login OK
    200 Type set to I.
    250 Directory successfully changed
    local: /var/www/web12/web/php-site-monitor/files/test_file3.mp4 remote: /var/www/web12/web/php-site-monitor/files/test_file3.mp4
    227 Entering Passive Mode (204,10,171,131,196,196)
    451 Can't create file "test_file3.mp4".
    221 Good-Bye
    Notice these lines:
    Code:
    local: /var/www/web12/web/php-site-monitor/files/test_file2.mp4 remote: /var/www/web12/web/php-site-monitor/files/test_file2.mp4
    227 Entering Passive Mode (204,10,171,131,195,230)
    451 Can't create file "test_file2.mp4".
    In the loop the script is trying to upload the file via FTP with the whole local path. And since that path does not exist on the FTP server, I'm getting the error creating the file.

    How can I fix this on the code?

    Also, what do I need to add in that shell script to detect if the file upload failed?

    Thanks

  11. #9
    Junior Member
    Join Date
    Jul 2010
    Posts
    5
    Thanks
    0
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Default

    Changing the code to this, does the work:

    Code:
    #!/bin/sh
    
    # Configure variables at the begining of the script
    MONITOR_DIR=/var/www/web12/web/php-site-monitor/files
    TMP_DIR=/var/www/web12/web/php-site-monitor/files/tmp
    SUCCESS_DIR=/var/www/web12/web/php-site-monitor/files/success
    FAILED_DIR=/var/www/web12/web/php-site-monitor/files/failed
    FILE_EXT=mp4
    
    # FTP variables
    FTP_HOST=upload-ftp.simplecdn.com
    FTP_USER=myuser
    FTP_PASS=mypass
    FTP_REMOTE_DIR=/
    
    cd $MONITOR_DIR
    
    for f in *.$FILE_EXT; do
    echo "Uploading " $f
    ftp -inv $FTP_HOST <<EOF
    user $FTP_USER $FTP_PASS
    binary
    cd $FTP_REMOTE_DIR
    put $f
    bye
    EOF
    #mv $f $SUCCESS_DIR
    done
    Notice the "cd $MONITOR_DIR" before the loop.

    However I'm still in the need to know how to detect if the file uploaded or failed, based in this implementation.

    Any ideas?

  12. #10
    Senior Member monk's Avatar
    Join Date
    Jan 2005
    Location
    Tibet
    Posts
    643
    Thanks
    5
    Thanked 43 Times in 38 Posts
    Rep Power
    15

    Default

    After a ftp returns, $? gives the exit status of the ftp command.
    Code:
    ftp -inv $FTP_HOST <<EOF
    user $FTP_USER $FTP_PASS
    binary
    cd $FTP_REMOTE_DIR
    put $f
    bye
    EOF
    
    if [ $? -eq 0 ]
    then
    	echo "Uploaded - do something"
    else
    	echo "Failed - do something"
    fi
    May the force with you!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 15
    Last Post: 13th December 2009, 01:54 AM
  2. Linux Shell Script to upload log files to Windows XP based FTP Server
    By stefanuscyberia in forum Shell scripting
    Replies: 6
    Last Post: 18th November 2009, 01:29 PM
  3. Shell script to monitor syslog.log
    By spabba in forum Shell scripting
    Replies: 1
    Last Post: 24th March 2009, 05:08 PM
  4. Replies: 1
    Last Post: 25th October 2007, 01:44 PM
  5. i need a script to delete one folder with files on my ftp
    By silver_ch in forum Shell scripting
    Replies: 1
    Last Post: 27th March 2007, 10:15 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •