nixCraft

Hello,

I have always thought that /tmp Folders and files were precisely for temporary use. Many of which are deleted on the shutdown or reboot.

I deleted some folders today, which I believe were made during backup processes, snap shots - some were days old, since the server had not done a reboot for quite some time.

I have just had the worrying experience to realise that after having deleted some of the /tmp Folders the corresponding folders on the NAS were apparently deleted also !
The amazing thing is that the deletion was so quick on the network that it must have been done in at least a 1/20 of the time it took to create the remote folder. In this case the folders names started as mntXXXXX etc. and seemed to be copies [left over] of the data that I had sent to the NAS in previous successful backups.

The folders in theory were not hard linked
I would like to know if anybody has ever had a similar experience and if it is well known that this can happen because I fail to see why deleting temporary files related to past backups should also delete the remote files on a NAS ... without warning.

Thanks for any suggestions

I'm not sure if I got your question.. Do you store both data and backup on NAS? Is /tmp mapped to nas?

Yes, delete operation over NAS can be very fast. You will only notice difference for large size files and overloaded network. NAS disks rotate at higher speed and works in RAID to improve I/O.

NAS folders are mounted using NFS or SMB or other supported protocol. If you delete a file from mounted directory, it will get deleted. Backup is only solution. Take regular backup.


No it should not delete files but I can think of few possiblities:

• /tmp is mapped wih nas over NFS or CIFS.
• /tmp is monitored using script and events. A cutomemade script watches the filesystem for specified changes and executes the relevant commands.
• Linux also allows to map same shared nas directory. So you can have it mapped to /tmp and /data.. run mount command to see connection.

HTH

__________________
Vivek | My personal blog
Linux Evangelist
+ ADD [SOLVED] thread prefix to your thread when your problem is sorted out by editing your thread.
+ Always use CODE tags for posting system output and commands!

Hello,
Thanks for the quick reply.
This business could have been a tragedy as you can imagine and I want to understand how it happened so as to avoid it in the future.

No the data (commercial analysis database is on SUSE 10.3 Enterprise Server, the NAS on the same network is a simple Buffalo TeraStation.

In this case it was an SMB connection, this NAS does not support NFS.

Please explain how cancelling a file or directory in the /tmp directory can wipe out the corresponding NAS backed up file or folder.
What logic does that follow ? I mean I do not understand why an App would make folders in the /tmp directory, the contents of which are of course prone to deletion, and hard link them to the remote area which contains the copied data so that the data could be accidentally eliminated.
In fact I will have problems restoring the data.

Thanks for confirming that it should not delete the remote files.

I believe I had 'killed' the app process before the event.
I ran a mount and I saw no reference to /tmp/XXX
On the other hand I did notice processes referring to CIFS.
But here again I did not map anything to /tmp.
I connected to the remote the remote destination folder using a SMB connection and in fact the App used an SMB connection to complete its backup.

Any other ideas or suggestions.
The alarming thing here is that if I reboot and the /tmp subdirs are cancelled while the NAS is online I could find that the NAS is emptied !
Seems absurd that a professional App created a situation like this.

[I will remain online for about 40 minutes Thanks]

Hi nicephotog,

Thanks for taking interest.
Could you explain a little more, I am not sure if I understand your post.
eg: used by point of the switch ?
... links inside the /tmp folders you deleted rhat point to the actual directories recursively ?
etc.

BTW, How would you go about detecting the symptoms you mention ?
Just to note: The application that was being used to backup the data [and the Server] to the Nas was a Backup SW which is not only quite well known but also quite often used by sys admins.
It not only backs up data but can also clone a computer using a type of Snapshot technology.

If you read my original post you can imagine how concerned I am due to the enormity of the consequences.
One could envisage situations where this could happen even without human intervention.

Thanks

Hello nicephotog,

I apologise for this belated reply.
Thank you for the follow up

1. You talk about switches in the command line and commands which could use a switch to follow a link and act on it.

> If the behavior was to follow or create linked "SYMLINK" directories

OK this is plausible but I fail to understand why a SW Backup App, which should be designed to save and protect data, should leave links of this type around in a directory, in particular /tmp and have commands/processes still in memory and able to delete remote data without warning.

If the behavior was to follow or create linked "SYMLINK" directories or symlinks in the directory for the command issued then it would destroy not only during the current delete command issued on the folder but everything within the symlinked folders in the folder being deleted.

Without any warning and used by a backup SW ?
Correct me if I am wrong but usually Symlinks are soft links and deleting the link does not delete the remote data. It creates an orphaned link.
Therefore it would seem that there was some special command which was active and was an error and a hefty one or there is something else which I have not yet understood.

"Symlinks have two main behaviors", 1. to attain a direct channel to another folder
2. Two imply the contents of a symlinked directory is part of the folder containing the link as though it were in it physically. [note: rightclick the folder and look at the permissions , the special settings "sticky" ].
"Symlink" term is short for Symbolic Link a file/folder icon usually has a small arrow curling from the bottom right corner to show it is a link.
...Recursively... Is when you have a folder present with a link to another folder thathas a link back to the original folder in it somewhere, ...is the easiest way to explain that.

When issuing a command to create a symlink you can use aswitch thatwill control some of that behavior,also when deleting a symlink file, you need to name to it to "not be recursive" when you delete it.

Usually symlinks [ln -s] as defined do not by definition alter the folders/files to which they refer or at least not on their own. Or am I missing something ?
Of course a custom script could do anything if desired; however my point or concern is that it seems absurd to have a SW which creates a situation where the deletion of folders or files in the /tmp can remove the very files which it is meant to protect/save/backup on remote locations.

You should email the programmers of your backup application and tell them what had occurred, its probably the way these folders are set.

In fact this is planned however I am hoping to try and recreate the situation in the lab so that I can understand what really was/is happening.

If you have any comments I would be be interested in reading them
Thank you

...

OK, I will look into this and, should I be able to recreate the phenomenon in the Lab, I will report the details to you.

Thanks for the help

Hi,
I have not as yet mentioned the name of the SW due to the fact that I have not had enough time to recreate a Lab Case of this and demonstrate that this "error" is due to the SW alone.

Since I believed, in the beginning, that the SW was first class I would not wish to make the mistake of unjustly attributing the cause of this disaster to the SW house.

However if it is the cause, as it has seemed to be, then it would be my intention of warning SysAdmins of this fact.

It should be noted that this incident occurred shortly after having had to terminate/interrupt one of its processes. This may not be the cause alone but could be of importance to some extent since, in a general case, admins [and/or other tasks/scripts] may terminate processes and thus one might be at risk if the certain sub folders of /tmp were altered.

In the meantime you will find a PM with details.
Thanks