06 January, 2013

Merge log files from multiple servers

If you have some similar servers (e.g. balanced EC2-instances) and need to analyze log files from them - it is possible to automatize downloading and merging all these files via shell script like this:

#!/bin/bash

LOG_DIR=log-`date +%Y.%m.%d_%H.%M.%S`

mkdir -p $LOG_DIR

scp -C ec2-user@node1.server.com:/log/file.log $LOG_DIR/file1.log
scp -C ec2-user@node2.server.com:/log/file.log $LOG_DIR/file2.log
scp -C ec2-user@node3.server.com:/log/file.log $LOG_DIR/file3.log

cat $LOG_DIR/*.log | sort >$LOG_DIR/files.log


Also it is a good idea to use ssh-agent to keep private keys.

No comments: