nixCraft Linux / UNIX / Shell Scripting Forum

nixCraft

Linux / UNIX Tech Support Forum

Shell script download mp3 files in bulk from a site

This is a discussion on Shell script download mp3 files in bulk from a site within the Shell scripting forums, part of the Development/Scripting category; Hello I want to wget a lot of files from another server to my server The remote server is just ...

Register free or login to your account to remove all advertisements.

Go Back   nixCraft Linux / UNIX / Shell Scripting Forum > Development/Scripting > Shell scripting

Shell script download mp3 files in bulk from a site Shell script download mp3 files in bulk from a site

Linux answers from nixCraft.

Register FAQ Members List Social Groups Calendar Forgotten your password? Mark Forums Read

Shell scripting You can discuss the shell scripting, request shell scripts and scripting techniques

Reply

 

Thread Tools Display Modes
  #1 (permalink)  
Old 16th December 2008, 02:30 PM
Junior Member
  
Join Date: Sep 2007
OS: CentOS
Posts: 21
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0
vivekv is on a distinguished road
Default Shell script download mp3 files in bulk from a site
Hello

I want to wget a lot of files from another server to my server

The remote server is just having the folder and files

example , HERE

You can see a folder there, and 5 or 6 files inside it.
I just want to copy those files. When there is a large number I cant wget it manually

I searched here but cant find a code to suit my need

Any help is appreciated

Thanks
Vivek
Reply With Quote
  #2 (permalink)  
Old 19th December 2008, 04:13 PM
nixcraft's Avatar
Never say die
  
Join Date: Jan 2005
Location: BIOS
OS: RHEL
Scripting language: Bash, Perl, Python
Posts: 3,195
Thanks: 13
Thanked 394 Times in 292 Posts
Rep Power: 10
nixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond repute
Default
Try something as follows
Code:
lynx --dump http://www.onlinetamilmp3.com/DW/LATEST%20HINDI%20RELEASES/Rock%20On/ | grep .mp3 | awk '{ print $2}' | grep http | xargs wget
Keep in find webmaster may not like this kind of scripting as it can put load on their server.
__________________
Vivek Gite
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
 Do you run a Linux? Let's face it, you need help!
Cricket & IPL News Blog
Reply With Quote
  #3 (permalink)  
Old 19th December 2008, 09:30 PM
Junior Member
  
Join Date: Sep 2007
OS: CentOS
Posts: 21
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0
vivekv is on a distinguished road
Default
You saved a lot of time ! Thanks for your kind help.

Now, there is something added to this .

Consider this , FOLDER

This folder has lot of sub folder. What I want to do is to , just duplicate the entire directory structre here.

I am in need of a shell script which actually do the following

1. Get the folder name from the site
2. Mkdir the folder here on our server
3. cd folder
4. Download all the mp3s from the site

I am not a good shell programmer, any help is highly appreciated

Thanks
Vivek
Reply With Quote
  #4 (permalink)  
Old 19th December 2008, 09:34 PM
Junior Member
  
Join Date: Sep 2007
OS: CentOS
Posts: 21
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0
vivekv is on a distinguished road
Default
By running this command

Code:
lynx --dump http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/ | grep / | awk '{ print $2}' | grep http
I get the following output

Code:
http://www.onlinetamilmp3.com/DW/
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/.snifthumbs
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/A%20Aa%20E%20Ee
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Aanandha%20Thaandavam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Aayutham%20Seivom
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Abhiyum%20Naanum
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Adadaa%20Enna%20Azhagu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Aegan
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Alibaba
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Arai%20En%20305il%20Kadavul
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Azhaipithazh
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Dasavadharam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Dhaam%20Dhoom
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Dhanam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Durai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Ghillidaa
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Guru%20En%20Aalu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Iyntham%20Padai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Jeyam%20Kondaan
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Kadhalna%20Summa%20Illai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Karuvarai%20Pookal
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Kathikappal
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Kee%20Moo
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Kodaikanal
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Kuruvi
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Kuselan
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Maanavan%20Ninaithal
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Mahesh%20Saranya%20Matrum%20Palar
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Meipporul
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Muniyandi%20Vilangiyal%20Moondramandu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Naan%20Aval%20Adhu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Nesi
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Nettru%20Indru%20Naalai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Newtoninn%20Moondraam%20Vidhi
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Pachai%20Niramae
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Pandhayam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Pandi
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Paththu%20Paththu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Perumal
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Poi%20Solla%20Porom
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Poo
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Poova%20Thalaiya
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Pulan%20Visaranai%20-%20II
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Raman%20Thediya%20Seethai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Rasi%20Engal%20Nalla%20Rasi
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Rasikkum%20Seemane
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Sakkarakatti
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Saroja
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Sathyam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Seval
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Silambattam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Sindhanai%20Sei
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Subramaniyapuram
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Suriya
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/TN%2007%20AL%204777
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Thalaikeezh
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Thee
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Thenavettu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Thiruvannamalai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Ulliyin%20Osai
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Vaaranam%20Aayiram
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Vaitheki
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Vannathu%20Poochi
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Vennila%20Kabadi%20Kuzhu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Villu
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Yean%20Indha%20Mounam
http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/Yennai%20Theriyuma
Reply With Quote
  #5 (permalink)  
Old 21st December 2008, 04:00 PM
nixcraft's Avatar
Never say die
  
Join Date: Jan 2005
Location: BIOS
OS: RHEL
Scripting language: Bash, Perl, Python
Posts: 3,195
Thanks: 13
Thanked 394 Times in 292 Posts
Rep Power: 10
nixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond reputenixcraft has a reputation beyond repute
Default
Try it as follows:

Code:
LIST=$(lynx --dump http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/ | grep / | awk '{ print $2}' | grep http)

for u in $LIST; do lynx --dump $u | grep .mp3 | awk '{ print $2}' | grep http | xargs wget; don
__________________
Vivek Gite
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
 Do you run a Linux? Let's face it, you need help!
Cricket & IPL News Blog
Reply With Quote
  #6 (permalink)  
Old 21st December 2008, 04:24 PM
Junior Member
  
Join Date: Sep 2007
OS: CentOS
Posts: 21
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0
vivekv is on a distinguished road
Default
Quote:
Originally Posted by nixcraft View Post
Try it as follows:

Code:
LIST=$(lynx --dump http://www.onlinetamilmp3.com/DW/LATEST%20TAMIL%20RELEASES/ | grep / | awk '{ print $2}' | grep http)

for u in $LIST; do lynx --dump $u | grep .mp3 | awk '{ print $2}' | grep http | xargs wget; don

But this will download all mp3 files to my server in a folder. It wont create any folder and put the mp3 files just like the remote structure.

What I mean is, I want to first create a folder here and <cd> to that folder and then download files ?
Reply With Quote
  #7 (permalink)  
Old 17th April 2009, 11:33 PM
Junior Member
  
Join Date: Apr 2009
OS: Centos
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0
sachin4sachi is on a distinguished road
Default
Thanks
Reply With Quote
Reply

Tags
awk, grep, lynx, shell scripting, wget, xargs

« BASH Script To Send Email When Something Fails in The system | Shell Script To Run Program at Login »

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off

Similar Threads

Thread Thread Starter Forum Replies Last Post
Shell Script: List Empty and nonempty (zero byte) files jawII Shell scripting 1 1st December 2008 11:19 AM
Shell code for Restoring cpmove accounts(bulk amount) vivekv Shell scripting 2 3rd December 2007 09:56 PM
Shell script for automatic conversion of files in tar files kasimani Shell scripting 2 8th February 2007 03:45 PM
shell script to open log files and check for faults trueman82 Shell scripting 1 23rd November 2006 02:35 AM
shell script that parses multiple log files and checks for a Anonymous Shell scripting 1 15th November 2006 09:38 PM


All times are GMT +5.5. The time now is 08:53 PM.

Contact Us - Home - Archive - Privacy Statement - Terms of Service - Top

Powered by vBulletin® Version 3.8.6 - Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.3.2
©2005-2010 nixCraft. All rights reserved

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40