NeIC Conference 2013: Report on "Workshop: Security"

Introductory Talk on Forensics (Leif Nixon)

The aim of the workshop is quick and dirty forensics using tools commonly available on systems.

How is an intrusion discovered?

• monitoring alarm, not only from security monitors
• external alert like a phone call or an email
• anomalous system behaviour like cron job which stopped working

When discovered, look at things like system logs, command histories, outpot from ps, top, netstat, lsof, etc. Do not run rpm -Va, as this alters time stamps, and increas the risk of overwriting valuable information. Also, do not reboot, kill suspect processes, or delete malicious files.

At the point you discover the intrusion, take a break and think things through. Now, there are two paths to follow. Depending on site policy, you want want or be required to contact the police. In that it may be necessary to hand over the investigation to the police or an expert. The policy may be a black hole, but reporting incidents may still have a positive effect to increase awareness, and some cases are in fact investigated.

Are there reasons not to report the incidence? Some sites does in fact have a policy against it, unwise as it may seem. You may also want to restore the service as as quickly as possible to a secure state. But don't just reinstall the system.

Before bringing the system back to a secure service, the following should be determined:

• How did the intruder get in?
• When did it happen?
• What have they done to your system?
• How can you stop them from compromising the system again?
• What other sites could the attacker have compromised?

Careful and thorough forensic investigation is hard to perform. A quick and dirty forensic may be enough. Sometimes in fact better since it's faster. Especially, if multiple sites are involved, one site doing a thorough investigation may be a bottleneck, e.g. when tracking an intruder backwards.

First actions are usually to check for open network connections, e.g. with netstat, and better cut and paste, rather than saving to the local disk. Now, unplug the network cable or introduce a router filter to block out the attacker. Make sure you are not interrupted while doing this. If you are discovered, the attacker may do bad things to your server and wipe evidence.

There are different kinds of data with different life times. You want to start with the least persistent. However, timestamps are so useful that it's generally good to secure them first of all.

The file system is made up of three levels of data: The directory entries link to inodes and inodes link to blocks. Inodes contain the following metadata:

• owner and group
• permissions
• type
• link
• mtime - the modification time which is the last change to the content
• atime - the access time which is the last time the file was read
• dtime - the time a file was deleted is recorded in deleted inodes of ext systems
• crtime - creation time, only available on ext4fs
• data block pointers

So, can we trust these timestamps and what do they tell us?

• mtime and atime can be set to anything.
• ctime is the most reliable. It can be changed by temporarily setting the system clock while modifying or with fsdebug, but it tricky to do without leaving other traces.

From these we want to generate a timeline. That is, collect timestamps for everyfile and sort them into a timeline. How?

The easy way: Run stat on every file on the mounted system. This can be done with a command like

   find / -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n

This is good for coaching an inexperienced admin, e.g. via phone or email. However, it means accessing all directories, and looking mean modifying. Also be careful where you store the data. If possible remount read-only or flip the atime option. Note that rootkits may hide files.

A more proper way: Bypass the normal kernel file system operation and dig out the data from an off-line image. This is slower an requires custom tools like the Sleuth Kit (TSK) from http://www.sleuthkit.org/. This can also find deleted inodes and directory entries.

What does the timeline really tell?

• ctime often tells when the file was created
• atime can tell when a file was read or a binary executed
• mtimes ca be useful because they can be modified: many tools try to preserve mtimes, so you can see timestamp from another system

But there are some complicating factors. You only see the last time timestamp, and they can be affected by things like: prelinking, updatedb, tmpwatch, and makewhatis. A histogram of timestamps can show on what time periods to focus. ls -i shows inodes numbers, which correlates with time due, esp for newer file system.

Deleted files are not erased from disk. In ext3, dentries are marked as deleted and directory lists are updated to skip the deleted entries, but the entries remains. Inodes are marked as available. For tecknical reasons, their block pointers are deleted, so we loose the links to the data blocks. Data blocks are marked as available, but their contents remain until reused. TSK can display deleted dentries and inodes. It's harder to retrieve the contents of deleted files. Sometimes it can be found though the journal using extundelete. Otherwise, use tools like strings and grep

Note that the file system is blocked, and since files often are smaller than the blocks, there is free space called slack space. This can contain data from deleted flies, as tell. It can also be used by an attacker to hide data.

Disk images can be created with

dd if=/dev/sda of=sda.img bs=512
dd if=/dev/sda1 of=sda.img bs=512

If the system is mounted, the images may be inconsistent, but still useful. Syncing first may help, but riskes loosing files.

Disk images can be investigated with TSK:

mmls -a sda.img                # lists partition table
mmcat sda.img 6 >sda2.img      # extracts a partition

You can mount images with

mount -o loop,ro sda2-copy.img mnt

Even a read-only mount updates the superblock of the file system, so working on a copy may be wise. If the mount fails, it may be necessary to mount rw breifly to let the journal update, or use the norecovery mount option. To save time, one can work with the partition in place:

fls -o <sector-offset> -r -m / sda.img >rootfs.body
mount -o ro,loop,offset=<byte-offset> sda-copy.img mnt

What about other data? If the system was root compromised, it may lye to us. Now that we have the timestamps, we can run rpm -Va to check the integrity of binaries. This command could be compromised, as well, but often it isn't. Now, copy in new versions of compromised commands or use alternative commands.

• Note that processes can change names.
• Look at PID numbers. System processes usually have PIDs in a narrow range. A high-numbered system look-alike is suspicious.
• Look at child processes. E.g. ping should not have a sub-process bash.
• Use netstat or lsof to check files and sockets.

When the process is found, look at it's memory:

• gcore dumps the process core.
• strings can find IP addresses and passwords
• A whole RAM dump may be useful, but has intentionally been made hard to do.
• Use strings -a on binaries to get all strings.
• Execute binaries under strace and ltrace but only in an isolated environment.

Trojans often obfuscate strings, but almost always by xor'ing with a single byte!

The intruder may have tried to cover his tracks, but there are multiple log files available, some which he might have forgotten:

• /var/log/secure - the SSH log
• /var/log/wtmp - binary DB of terminal sessions
• /var/log/btmp - binary DB of failed logins
• /var/log/lastlog - binary DB of latest logins per user
• /var/log/audit/audit.log - from the audit subsystem

Rootkits are somewhat problematic for the quick and dirty approach, as it puts a lot of faith on key system binaries. chkrootkit and rkhunter can find some common cases. TSK can usually detect files hidden by the kernel rootkits. It may be useful to compare the output of fls to that of find or ls.

When you are ready to re-enter production you should have a good idea what the intruder did so you can clean the system from malware and plug any holes the intruder might have exploited. If the system was root compromised, you can't be sure, so you almost always want to reinstall.

"Co-chair for WS Security I--IV" (Sven Gabriel)

This part is not reported here to the benefit of future participants.

The l33test team were Regulus which consisted of Bjørn-Helge Mevik, Jon Kerr Nilsen, and Thierry Toutain. Congratulations!

Discussion Points

The discussion was focused on the exercises, and thus also omitted here.

Lessons Learned

• Participants were happy with the session.