nixCraft Linux Forum

nixCraft

Linux Tech Support Forum

reading log files (performance issue)

This is a discussion on reading log files (performance issue) within the Shell scripting forums, part of the Development/Scripting category; Hi All, I am reading some logfiles and parsing data and printing to some textfile. Here is my code OLDIFS=$...


Go Back   nixCraft Linux Forum > Development/Scripting > Shell scripting

Register FAQ Members List Calendar Mark Forums Read
  #1 (permalink)  
Old 05-05-2008, 08:35 PM
Junior Member
User
 
Join Date: Apr 2008
My distro: Debian
Posts: 5
subin is on a distinguished road
Default reading log files (performance issue)

Hi All,

I am reading some logfiles and parsing data and printing to some textfile.
Here is my code

OLDIFS=$IFS
IFS='
' # just a newline, in single quotes
while read data
do
if [ "$data" != " " -a "$data" != "" ]
then
#Parsing the Frontend log file
MSISDN="`echo "$data" | cut -c1-10`"
HOUR="`echo "$data" | cut -c11-16`"
ID_SA_SOURCE="`echo "$data" | cut -c17-34`"
ID_SA_DEST="`echo "$data" | cut -c35-52`"
ID_VIR_PORTAL="`echo "$data" | cut -c53-70`"
NW_BEARER="`echo "$data" | cut -c71-74`"
TERMINAL_TYPE="`echo "$data" | cut -c 75-75`"
TRADE_MODEL="`echo "$data" | cut -c 76-105`"
HOUR=$LOGDATE$HOUR

echo $HOUR";"$MSISDN";"$ID_SA_SOURCE";"$ID_SA_DEST";
"$ID_VIR_PORTAL";"$NW_BEARER";"$TERMINAL_TYPE";"$T RADE_MODEL >> OFR_Processed_data.txt
fi
done < $TRACKING_LOGDIR/$listdata

In the log file my data is always fixed thats y ia m using cut to get the data.
This code is working perfectly, but performance vice its a big failure.
while reading a logfile with 1 lakh records its taking morethan 2 hours.
Can any one tell y it is taking this much time ? How can i alter my code in a better way ?

Thanks in advance
Subin
Reply With Quote
  #2 (permalink)  
Old 05-21-2008, 06:48 PM
Junior Member
User
 
Join Date: Feb 2008
My distro: .
Posts: 9
unSpawn is on a distinguished road
Default

If your format is fixed then you could start by substituting 'echo|cut'?:
Code:
while read DATA; do
 echo ${DATA:0:10} ${DATA:11:6} ${DATA:17:17} # etc, etc..
done
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads

Thread Thread Starter Forum Replies Last Post
PHP code performance zafar466 Linux software 1 04-22-2007 08:23 PM
Re-reading the partition table failed with error 16: Device chiku Linux software 1 12-07-2006 08:45 PM
regarding server performance lovewdhwa Linux software 1 09-21-2006 01:42 PM
Log Off Issue tkinsella Linux software 1 08-30-2005 03:03 PM
Need Help reading files mustang Shell scripting 1 07-22-2005 01:49 PM


All times are GMT +5.5. The time now is 12:48 AM.


Powered by vBulletin® Version 3.7.2 - Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
LinkBacks Enabled by vBSEO 3.1.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30