Library Guide

script for generic archiving in linux

script for generic archiving in linux

Below a script for archiving, compressing and deleting files from a logfile directory.
This script retrieves the directories from a config file in the /etc directory.
The configfile is /etc/archiveer_custom_dirs

The script can be scheduled in the crontab.

The only parameterized values at this moment are the directories(via the config file).
At this moment other parameters are fixed in the script, like
- mail adress for output
- interval value archiving files
- interval value compressing files
- interval value deleting files
- archive destination directory
- the extensions of the log-files (.txt, .log, .his, *.log.* )

The script monitors the directory from the configfile, it creates an archive directory(archief) in the logfile directory.
Then it moves all config files(with extensions .txt, .log, *.log.* to the subfolder achief when the logfiles are older than 14 days.
The files older then 21 days in the subfolder archief are compressed.
The files older than 180 days are removed from the archief subdirectory.

In the end the output is mailed.

#!/bin/bash
#
# shell script for archiving log files
# created : Jan van Overveld
#    date : 1 mei 2015
# 
# input param directory to monitor
#
# Define the directory which has to be archived.
# LOG_FILE_DIRECTORY="/opt/alfresco/tomcat/logs"
#
# voor debug
#set -x
#
execute_archive () {
  # $1 the folder to be archived
  # $2 the logfile for the echos
  if [ $# -ne 2 ]; 
  then
    echo "execute_archive needs 2 parameters" >> ${2}
    return $false
  fi
  #
  echo "   START voor directory ${1}  " >> ${2}
  #
  if [ ! -d ${1} ]; 
  then 
    echo "${1} is geen Directory !" >> ${2}
    return $false     
  fi
  #  
  LOG_FILE_DIRECTORY=${1}  
  #
  echo "   Create archief directory wanneer deze nog niet voor komt. " >> ${2}
  LOG_FILE_DIRECTORY_ARCHIVE=${LOG_FILE_DIRECTORY}/archief
  if [ ! -d ${LOG_FILE_DIRECTORY_ARCHIVE} ]; 
  then
    echo "   Directory ${LOG_FILE_DIRECTORY_ARCHIVE} wordt aangemaakt. " >> ${2}
    mkdir ${LOG_FILE_DIRECTORY_ARCHIVE}
  fi
  #
  echo "   " >> ${2}  
  echo "   Verplaats bestanden ouder als 14 dagen naar de archief directory(${LOG_FILE_DIRECTORY_ARCHIVE}). " >> ${2}
  find ${LOG_FILE_DIRECTORY} -mtime +14 -type f -not -path "${LOG_FILE_DIRECTORY_ARCHIVE}/*" ( -iname *.txt -o -iname *.log -o -iname *.his -o -iname *.log.* ) |
  while read fname
  do
    #stat ${fname} >> ${2}
    MODIFY_DATE=`stat --format=%y ${fname}`
    echo "      ${fname} - ${MODIFY_DATE} . " >> ${2}
    mv ${fname} ${LOG_FILE_DIRECTORY_ARCHIVE}
  done
  #
  echo "   " >> ${2}    
  echo "   Archiveer en comprimeer bestanden ouder als 21 dagen en nog NIET gearchiveerd (in de archief) directory(${LOG_FILE_DIRECTORY_ARCHIVE}) . " >> ${2}
  find ${LOG_FILE_DIRECTORY_ARCHIVE} -mtime +21 -type f -not -name "*.gz" |
  while read fname
  do
    MODIFY_DATE=`stat --format=%y ${fname}`  
    #stat ${fname} >> ${2}    
    echo "      ${fname} - ${MODIFY_DATE} . " >> ${2}    
    if ! gzip "$fname"
    then
      echo " !! FATEL ERROR !! FATEL ERROR !! compressing log $fname !! " >> ${2}
    fi
  done
  #
  # delete old zip files after one year
  echo "   " >> ${2}      
  echo "   Deleting files older then 6 months in folder. (${LOG_FILE_DIRECTORY_ARCHIVE}) " >> ${2}
  find ${LOG_FILE_DIRECTORY_ARCHIVE} -type f -name *.gz -mtime +180 | 
  while read fname 
  do
    MODIFY_DATE=`stat --format=%y ${fname}`  
    echo "      ${fname} - ${MODIFY_DATE} " >> ${2}      
    #stat ${fname} >> ${2}    
    if ! rm $fname
    then
      echo " !! FATEL ERROR deleting file $fname !! " >> ${2}
    fi
  done
  #
  echo "   EINDE voor ${1} " >> ${2}
  #  
  return $true
}
#
TIMESTAMP=`date "+%Y-%m-%d %H:%M:%S"`
LOG_FILE="/tmp/archiveer_log_file.txt"
CONFIG_FILE_NAME="/etc/archiveer_custom_dirs"
#
echo "Start archiveren. " > ${LOG_FILE}
echo "Datum en tijd : ${TIMESTAMP} " >> ${LOG_FILE}
#
#
if [ -r ${CONFIG_FILE_NAME} ];
then 
   while read p; do
      if [[ ${p:0:1} != "#" && -d $p ]];
      then
         echo "archiveer uitvoeren voor ${p}" >> ${LOG_FILE}
         execute_archive $p ${LOG_FILE}
      else
         if [[ ${p:0:1} != "#" ]];
         then
            echo "directory komt niet voor : ${p} " 
         fi
      fi
   done <${CONFIG_FILE_NAME}
else
   echo "Config bestand bestaat niet : ${CONFIG_FILE_NAME}" >> ${LOG_FILE}
fi
#   
#execute_archive "/root/testdir" ${LOG_FILE}
#execute_archive "/var/log/alfresco" ${LOG_FILE}
#execute_archive "/opt/alfresco/tomcat/logs" ${LOG_FILE}
#
echo "Eind archiveren. " >> ${LOG_FILE}
#
# MAILEN VAN DE RESULTATEN
#
HOSTNAAM=`hostname`
SUBJECT_TEXT="${HOSTNAAM} / ${TIMESTAMP}  (LOGS cleaned) "
#echo ${BODY_TEXT} | mail -A ${LOG_FILE} -s "Archiving, compressing and deleting log-files" jan.vanoverveld@krinkels.com
cat ${LOG_FILE} | mail -s "${SUBJECT_TEXT}" janvanoverveld@gmail.com
#
exit 0

download file now