Linux Sed Multiple Lines

Using sed to delete multiple lines from a file in linux

I remember how frustrating it was before I figured out the easy multiline sed command. This example is especially useful to clean up website hacks / injections - a "regular" single-line sed is fine to remove most of the common base64 injections that hit most of the files in the document root... But sometimes you come across JavaScript injections that are clearly "tagged" so there's an easy way to identify it in all injected files (even if there are 100,000+), but substituting part of the "signature" into your usual sed command just doesn't work.

Enter the multiline sed

There are two ways to do this, I'm going to go over the simpler one here, since if you are reading this, you probably need this information now:

Replace or remove multiple lines from a text file (the easy way):

sed '/FIRSTLINETODELETE/,/LASTLINETODELETE/d' /the/file/to/delete/from

*Lines can be regex, but make sure you get the whole line

Note: if you are cleaning script injections with a unique "tag" in the comment before and after the injection, just replace UNIQUETAG below with the random characters in the comment:

user@host [~]# sed '/<!--UNIQUETAG-->/,/<!--UNIQUETAG-->/d' /your/injected/file

This will output what the file will look like, but sed without -i will not modify the file.

If the output of the above looks correct, add -i after sed:

sed -i WILL modify the original file - please be sure your output is correct

sed -i '/<!--UNIQUETAG-->/,/<!--UNIQUETAG-->/d' /your/injected/file

"Regular" sed command, operates on a single line by default:

user@host [~]# sed 's/<!--UNIQUETAG.*UNIQUETAG-->//' /your/injected/file

This version will NOT work across multiple lines

Here's a test example that simulates an actual tag you may see in a javascript injection:

*Note: I'm working in Centos but this should all work the same on most Linux distros. Also, for completeness' sake the other and "proper" way to do multi-line sed is the N option, which allows you to do much more complex tasks. This example deletes the last 2 lines of a file:

user@host [~]# sed 'N;$!P;$!D;$d' fileName

More multiline sed examples

In over your head? Call a Developer for help with Linux, servers, code, websites, and more. Let us worry about the tech stuff while you move your business forward.

Links

• Sed linux examples
• Definitive Guide to sed: Tutorial and Reference
• Advanced sed commands
• Linux Cookbook
• Linux sed command

"Linux Sed Multiple Lines" continued here...

Howto: Disable SELinux on Centos

How to Disable SELinux in Centos Linux

Occasionally, you will have trouble with selinux on Centos and need to temporarily disable selinux. Here is how you change selinux settings in centos and other red hat linux distributions:

• Edit /etc/selinux/config (e.g. $sudo vi /etc/selinux/config)


• Find the line:
  
  SELINUX=enforcing


• • a) If you simply want to set selinux to permissive mode - which will still warn you when something would have been denied
    
    change to
    
    SELINUX=permissive
  
  
  • b) If you are sure you want to completely disable selinux
    
    change to
    
    SELINUX=disabled
  
  
  and save (in vi type [Esc] :wq)
  

  



• SELinux will be disabled after reboot. To turn selinux off immediately, without rebooting use:
  
  $ sudo setenforce 0

• If everything works, edit your /etc/grub.conf and change it from this:
  

  



• To this:
  

  

That is how to disable selinux on Centos. Note: there is no way to uninstall selinux from Centos. This is as disabled as selinux gets.

You usually will not need to disable selinux completely. This may be useful for troubleshooting, but I would recommend trying to disable IPv6 and disable your firewall first (service ip6tables stop; service iptables stop) and see if you have another problem. Disabling selinux can come in handy for various testing environments and may solve a few obscure problems with certain software.

Links:

• Linux Security Cookbook
• SELinux Wikipedia
• SELinux Sourceforge
• Centos SELinux HOWTO

"Howto: Disable SELinux on Centos" continued here...

How to Recover from a Hard Drive Crash

The day we all dread finally came - your (or your client's) hard disk crashed. The disk might be making funny noises or simply throwing errors when you boot. There is usually a whole bunch of data that hasn't been backed up, but it's not so important that you're willing to spend $4000+ to recover it (yes, data recovery does really cost thousands of dollars. If it doesn't, they probably won't go beyond what I show you how to do below, so you may as well try it yourself). Here is what you should do when you need to recover data from a crashed hard disk. Note: in many cases, the disk actually works fine (corrupt MBR or MFT, system file corruption, etc) - if you are lucky, you can literally "clone" the failed disk to a new one and boot into your old system!

Stop! Don't open, click, type, or touch anything. Turn off the computer by holding the power button or unplugging it. Unplug the computer.

Get a hard drive that is identical (preferable) or larger than the crashed hard disk.*

We are going to try recovering the data from the failed hard disk to the new one by using a bootable rescue CD. You can either add a second hard disk to the affected PC, or use another computer. If you use another computer, I would recommend you still plug in only the new disk and the one you are recovering data from, to avoid confusion.

• 0. Before you do anything, discharge your static well away from the computer, and put on a static wrist strap.


• 1. First, download and burn SystemRescueCD, or your favorite computer recovery toolkit. There are several others - in fact, most Linux LiveCDs will work, and you can easily make your own rescue cd also.


• 2. With all of your disks unplugged - unplug power and data cable - turn the computer on and go into Setup (look for the "Press DEL to enter Setup..." when the computer boots and press that key). Look for the "Boot Order" or similar option, and use the + and - keys (or PgUp/PgDn, it should tell you in the sidebar) to remove "hard disk" from the boot order, or at least move it to the bottom. Make sure CD or DVD is listed - preferably first.

 

• 3. Add a second hard drive  (as "primary" for IDE)
                Note: If your hard disk is physically damaged, making noise, or emitting smoke, do not leave it in the computer. While the computer is unplugged, very carefully place the disk on a static-free surface (like rubber - a mousepad might work), plug it in (may need a longer cable), get a powerful fan or an icepack (DO NOT cover up the little tiny air intake hole on top of the drive, or get the drive wet) and try to keep the crashed hard drive cool. In this case, you may have to make many short passes with long (hour+) breaks in between to let the drive cool down. Remember to turn off and unplug everything when you are not actively recovering data.


• 4. Power on the computer, quickly insert the Systemrescuecd, and Press a Key if prompted to boot to the CD. 


• 5. Identify the failed hard drive and the new disk that will hold the recovered data. Usually these will be sda and sdb (SATA) or hda and hdb (IDE), but make sure to check - you don't want to move the blank space from the new hard disk and wipe out the failed hard drive!
• 6. Use ddrescue to move the contents of the failed hard disk to the new hard disk.
          Note: this process is different depending on your kernel (and ddrescue) version. I am only listing instructions for 2.6.3 and later. These instructions will work for the latest Systemrescuecd. If you have to use an earlier kernel, please visit the Forensics Wiki for instructions. (-d is the same as --direct, and -v is the same as --verbose)
  
            First pass: get the data that is still intact (no retries, don't split)
  ddrescue --no-split --verbose /dev/sda /dev/sdb rescue.log
            Second pass: Retry errors 3 times, don't use the disk cache
  ddrescue --direct -v --max-retries=3 /dev/sda /dev/sdb rescue.log
            Third pass: Add retrim which will retry reads in a different order
  ddrescue -d -v --retrim --max-retries=3 /dev/sda /dev/sdb rescue.log
   

• 7. If you have successfully recovered the whole disk, you will want to check the disk for errors. Most operating systems will do this if you simply boot to the new disk containing the recovered data (unplug the old disk completely), but to be sure you may want to run the appropriate file system checker from the rescue disk (after rebooting with only the new hard disk in the system):
      FAT / NTFS         chkdisk /F /R C:\
      Ext2 / Ext3          e2fsck  /dev/sda1


• 8. If you haven't recovered all of your data, you can still try to mount partitions from the failed disk. The ro indicates "read only", while the noexec prevents anything on the disk from being executed, and the noatime prevents the file access time from being updates, so there really should be no disk writes.
  Try: mkdir /mnt/data; mount -o ro noexec noatime /dev/sdb /mnt/data
  If that doesn't work, try a partition at a time:
  
             mount -o ro noexec noatime /dev/sdb1 /mnt/data1
             mount -o ro noexec noatime /dev/sdb2 /mnt/data2
               ...


• 9. Then you should be able to see at least some of your files and copy them elsewhere using cp /mnt/data1/* /backup/folder/or/mounted/network/drive/ 

If you have not yet been successful, then you probably have to pay for professional data recovery or forget about that data. If you are feeling adventurous, or the data is extremely important but you can't afford to pay, you might want to try some forensic recovery tools. A good open source tool is The Sleuth Kit (make sure you get Autopsy with it). For a professional tool, consider EnCase. 

If your disk is damaged because of "Head-to-Disk Interference" or HDI, there is little chance of data recovery. This relatively uncommon drive failure is when the platters and/or head become off balance and the head physically grinds across the platters, "scratching" the disk much like a CD. If you experienced this you would know by the metal-on-metal grinding noise coming from the drive as soon as it tries to spin. That's data being completely destroyed. If you have a few hundred thousand dollars, you can recover the data on the non-damaged parts of the disk platters using various advanced electron microscopy techniques, but realistically you should probably move on with a fresh operating system on a new disk - why not give Linux a try?

More about mounting disks with the mount command

More about ddrescue

"How to Recover from a Hard Drive Crash" continued here...

Quick Tech Tips - Part 2

When faced with a computer you know nothing about, you will want to at least know the basic system information to avoid headaches. This is not only useful for troubleshooting, and applies to client machines also - can you tell me what processes are running on your system right now? So how should you approach an unfamiliar computer system? How do you get the system specs, logs, and other information you might need to fix a problem, pass on to a technician, or list in your asset log?

This is what I do to gather specs and system information before starting to work on a system that I know nothing about:

Windows


• Backup the system: Start->Run...->ntbackup


• Start->Right Click My Computer->Properties


• Start->Run...->taskmgr


• Start->Run...->msinfo32


• Start->Run...->msconfig


• Start->Run...->eventvwr


• Start->Run...->cmd; ipconfig /all


• Start->Run...->gpedit.msc


• Start->Run...->regedit

Linux


• Backup the system: tar or rsync


• dmesg


• uname -a


• free


• top


• lsmod


• lspci


• cat /proc/cpuinfo


• cat /var/log/messages


• vi /etc/sysctl.conf


• cd /etc/sysconfig; cat [things that you want info about] (e.g. cd /etc/sysconfig; cat network)


• ifconfig /all


• You will find much more information browsing around the /proc virtual filesystem, so e.g. to get the current max threads in the kernel, use: cat /proc/sys/kernel/threads-max

Am I missing a tech tip you use regularly? Add it in the comments below!

"Quick Tech Tips - Part 2" continued here...

Quick Tech Tips - Part 1

This is a series of short commands or scripts that help you do a simple Windows or Linux task. Basically a collection of cool hacks I find around the 'net.

Windows - Resync time on Windows boxes:

• If you see errors related to time, or your clock is off, do
• w32tm /resync
• If that didn't work, try the steps at Windows IT Pro.

Linux - Find what packages are required for an rpm (yum)

• rpm -qR package.rpm > requires.log
• OR
• rpm -i --test package.rpm > requires.log
• Then, if you are using yum, you can:
• yum whatprovides required.so.6

"Quick Tech Tips - Part 1" continued here...