Do not use, republish, in whole or in part, without the consent of the Author. TheTAZZone policy is that Authors retain the rights to the work they submit and/or post…we do not sell, publish, transmit, or have the right to give permission for such…TheTAZZone merely retains the right to use, retain, and publish submitted work within it’s Network

Code: Select all

Tiger Shark from Antionline has kindly given his permission for his tutorial to be hosted at The Taz.

You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=259056

Enjoy


Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.

This is a story. It’s fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the “clean-up” and mitigation of the breach.

Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.

—————————————————————————————————————–

During his mammoth reading task Dirk determined that he had found a reverse telnet connection by using Netcat to make the connection out through the firewall on port 80 and then spawning a command prompt that could be controlled from the remote computer. He had further decided that what he had seen in the transaction logs was a form of SQL injection that was used to enumerate the tables and their content until an administrative login name and password could be returned. From there he concluded that the rest was trivial. He looked at his watch. It was now 9:30pm. He walked down to Mike’s office and found him still there on the phone. He waited for several minutes while Mike finished his call.

“You know, I think we have it. I won’t go into the details but I know how he got in, what he did when he was here and thus how he stole the information. I also think I have the IP address of his computer. Did you hear from him again yet?”
“No, and to be honest I don’t think we will before tomorrow”
“Then maybe we have time to call the authorities and have them deal with him.”
“You think so? Er…. what does this IP address thing give us?”
“It’s the unique address of the computer on the internet that he carries out the attack from”.
Mike paused for some while. “You think we can pull it off?”
“It wouldn’t be up to us but we need to give the FBI as much time as we can. I’d say it’s now or never. I have the phone number for the local office right here.”
Mike thought again and finally, resignedly said “No. Sorry, this stays in-house.”
“It’s your company Mike….. Your decision”

Dirk was a little angry as he walked back to his office. He wanted a piece of this thief that had come into his network and made him look bad. “Well, he’s not coming back in.” he thought as he diverted over to Amy’s office switched on her computer and deleted the offending files and the scheduled task. “There, screw you… asshole”.

Earlier Gary had set up the monitoring for any internal traffic directed at port 80 and had sat back to think about how he was going to lock out the access when he needed to. He knew he had eighteen machines here and one other in the Cincinnatti office. How would he find out how many other boxes there were? His mind was wandering and the two thought processes collided. “Oh Duh….” he thought, “Why not just do an ‘Al’? Scan the subnets for port 80 after six in each time zone. They’ll show up on my monitor but I’ll be able to recognize mine by the source address. Perfect”. He was left with one other problem. How to determine if there were more than one access point. “The firewall logs are going to show me that.” he thought as he picked up the phone to Cincinnatti. He spoke to the admin there and requested the command line for the scheduled job on Dan Ereg’s machine. A few minutes later the phone rang.

“Gary speaking”
“It’s Tim in Cincinnatti. I have your command line.”
“I just need the IP address it connects to.”
“No IP address, it’s pointed at a domain name, al.attacker.com”
“Ok, he probably wants to be able to move, thanks”

After putting the phone down he went to the Secure Logging System and filtered the previous month’s logs for the IP address he had resolved through nslookup. There they all were. “Damn, this guy sucks. He’s so predictable. Always port 80, always 6:00pm in the time zone and only three machines. Well that will make the cleanup easier” he thought. Just to be sure Gary then filtered the logs for the previous month against the internal IP addresses and looked at the transactions immediately after six pm each night. He found they all pointed to the same address, al.attacker.com except one, the very first. “Noooo… It can’t be.”, he thought “He can’t be this frigging dumb…. Can he?” He quickly ran a Whois against the IP and found it was a common high speed ISP. “C’mon now.”, he thought, “Let this be my lucky day”. He opened google and entered the IP address. “Oh Baby, twelve hits”. They were mostly abuse reports from different locations. “Sweet, a pattern, this guy has been practicing from home and got himself noticed before”. The eighth one down really caught his eye. It was an email to a list which, (as many do), contained the headers. There was an email address too. Better yet the email address showed as a real name, <Joshua Albin>. “Well what have we here…. It’s ‘Al’…” Gary laughed out loud as he reached for the company phone book. He quickly looked up the number for the President of the Board and dialled it.

“The President’s office, Julie speaking, how may I help you?”
“Er, yeah, hi, it’s Gary Cunez, Corporate Computer Security Manager. I really need to talk to the president, like, er…., right now, is he available?”
“One moment please” as Julie placed him on hold.
A few moments later he was talking to the President.
“Gary, This is Bill Smoltz, the president, what do you have for me?”
“Sir, I’m pretty sure I have ‘Al’. His name is Joshua Albin. With some assistance from his ISP, HighSpeedAtHome.com, I think we can give the authorities an address.”
“How did you manage that. I thought these people did everything they could to hide themselves?”
“I won’t go into detail but he is pretty sloppy and made a single, rather large mistake. It took only a couple of minutes of digging which showed a pattern of abusive behaviour and, more importantly an email address.”
“So what do you want me to do?”
“Let’s bring in the FBI. With this amount of information they could have him in hours and he would be out of our hair. Not to mention the fact that if he’s only asking for ten thousand he’s probably running the same trick against others. We can help ourselves and others.”
“It’s a bit late Gary. Two hours ago a company statement was Fedex’ed overnight to all our investors stating that their accounts were frozen without each transaction being accompanied by a code issued with the statement and that the company would not deal with blackmailers or other fraudulent activities now or in the future.”
“Sir, that’s ok. How do you think the investors would feel if they received a second statement within twenty four hours telling them to relax, we caught him? I have him Sir, I know it.”
“You do have a point.”, Bill hesitated, “Ok, you call the FBI and give them what you have. I have some more calls to make now. I’ll get back to you”
“Thank you Sir. I really feel good about this”

Early the next morning Agent Hicks sat for two hours with Gary as he went over the evidence he had collected. Gary finished up by showing him the first IP address, the ISP, the Google results indicating a pattern from the IP address and finally the email.

“I think you have him there Gary my man”, Agent Hicks smiled, “You are even luckier than you think. I’ve worked with HighSpeedAtHome before, several times. They are great record keepers like yourself. This may only take a phone call or two to start my little ball rolling.” He grinned broadly.
“Phew, I’m glad. It’s a gut feeling that this is the guy but everything points to the owner of that IP being a bad kid in cyberspace.”
“I can’t argue with you there. Can I use your phone?”
“Yeah”

Agent Hicks made a couple of calls. “Ok, I know who I need to talk to now,” he said, “Let’s see what HighSpeedAtHome have for us” as he dialed the number he just noted down.

“Yeah, Hi, Marvin Brenner please”
“Yes, I’ll hold”

Hicks waited.

“Yeah, Marvin Brenner? Hi, This is Agent Hicks of the FBI, you should have been informed that I would be calling”
“Good. I need to know if there have been connections between the following two IP addresses in the last month. Can you do that?….. Good”. Agent Hicks read off Gary’s Cincinnatti IP address and that of Joshua Albin. A minute or so went by and he began listening again.
“Good…. Ok, can you confirm that there were no other connection attempts to the second address I gave you just before or after that connection was made”. Another long pause ensued.
“Ok, the second IP address had two attempted connections on port 1433 five minutes after the connection in question. Can you look at the source address of the port 1433 attempt and tell me if this was a single instance or part of a scan?”
“Ok, so you are saying that three weeks ago today at 6:00pm almost exactly the first IP address connected to the second on port 80 and that apart from a confirmed portscan for an SQL server from an unrelated source which touched the second IP address there were no other connections in or out for more than an hour…. Great, I want you to hold the logs because there will be a subpoena for them before you finish work. One final thing, a yes or no question, is the subscriber’s name Albin…..” another pause, “William Albin. Great, I’ll be seeing you soon, Bye.

Agent Hicks hung up and turned to Gary.

“Nice work…. It’s ‘Al’ all right, no question. The dumbass made the first connection go to his home. I have to go, I’ll be in touch”

It was 5:30 pm and Joshua was bored. He’d spent most of the day trying to work out how he was going to get the money without giving himself away and still wasn’t any better off. Since leaving school two years before and deciding college wasn’t for him he had wandered, unsuccessfully, from job to job. His current employment was the graveyard shift cleaning a large hotel’s kitchen and he hated that too. The doorbell rang downstairs. “Screw it,”, he thought, “the old man can get it…. he needs the exercise”. He heard the door open and some muffled conversation.

“Josh, Can you come here a minute” William Albin called up the stairs
“What now dad, I’m busy”
“It’s a girl to see you son”
“Er, just a minute”

Agent Hicks nodded thanks to William for not alerting his son. Joshua appeared round the corner and was confronted by two “suits” and two badges.

“FBI Joshua. Please be calm, we just need to ask you some questions”. Before Joshua could speak Agent Hicks reminded him of his right to silence and a lawyer. Joshua’s only comment was to his father, “I wasn’t going to take the money Dad, I was just showing them how vulnerable they are…. Dad….”

Agent Hicks colleague took Joshua to the waiting car while Agent Hicks went upstairs and removed a computer and a laptop from Joshua’s room.

Gary took the phone call from Agent Hicks.

“We have him and his computers. No doubt it was him, he partially confessed to his father when we arrested him. I’m going to be busy for a while, I’ll get back in touch when I have what I need. Just make sure that your evidence stays clean, ok”
“Yeah, no problem, it stays where it is…. and thanks”
“No problem, you made our job real easy, bye.

Gary called the President again and informed him of the developments. The second statement was already prepared and was sent out to all the investors.

An hour later in Seattle Dirk and Amy were just ordering dinner while he tries to explain the technicalities of how the attacker was causing her machine to connect to his and then have him control it. He felt pretty good about the whole thing. He had found the compromise, worked out how the attacker was doing it and had even found the attackers computer. Yep, he was “The Boy” and he was going to leverage that all he possibly could with Amy tonight.

Mike sat in his office at TFCU. “It’s getting late he thought” as he looked at the clock in the system tray of his computer. He noticed it change from 5:59 to 6:00pm. What he didn’t notice was the flicker of the drive light under the desk as his computer sent a DNS request for al.attacker.com followed by a SYN……..

—————————————————————————————————–
The Synopsis

Subsequent investigation and questioning of Joshua proved that the initial breach took place through a malicious script disguised as an enticing spam message sent to select members, (usually high ranking company officers), that insinuated itself into the local security zone of the machine and downloaded Netcat disguised as msbackup.exe and created the scheduled job for 6:00pm each evening to set up the reverse connection through port 80 on the firewall thus making it hard to detect and even harder to stop. From there Joshua had used other tools to elevate his privileges and compromise numerous machines throughout his “conquered” networks. It was then simply an issue of determining where the financial information was kept and how to get the required authentication information or use other methods such as SQL injection to get the data he required.
—————————————————————————————————–
The Lessons

Dirk’s a nice kid and not a bad admin, but he was utterly unprepared. It wasn’t his fault. We’ve all seen it, whether it be a new job or simply going to help a friend. The network that was never built with a cohesive plan in mind. It takes months to learn it and maintaining it at the same time means things simply “go by the wayside”. Dirk found himself in a stressful situation which he had no experience with and, worse yet, had no plan to try to address the problem. People say that “information is power” and it is true. Logfiles are information. Therefore logfiles _are_ power. Even if you are unprepared for a situation a comprehensive logging system can allow you to “muddle through” without the nicely documented plan your boss would like to see that will still probably contain “holes”. The holes can be filled by the information in the logfiles. As hard as Dirk tried his efforts were stymied by the lack of information. That lack of information coupled with a lack of understanding of how the attacker works led him to believe that he had a clean network, that he knew what had occurred and where the attacker was attacking from. The reality is he was left with a dirty network, he had no way of knowing that other machines were compromised, he didn’t understand that Joshua wasn’t attacking him from his own computer and he didn’t know that Amy’s machine was a “jump off” point inside his own network, (though in this case Joshua did use her machine to get the data he needed, logs may have shown him a different picture).

Gary has an advantage over Dirk. His company employs him to do the security. He’s been there for a while and he’s built his security architecture himself. He monitors the network, he logs it heavily and he wrote his procedures while he had time to research them. He also watches what the potential attackers can do and adjusts his systems accordingly. So when the “bad thing” happens he can move in, confident that unless the attack is extraordinary he has a good chance of piecing it all together. Gary ended up with a clean network, sufficient information to put Joshua in jail for three years and a handsome pay raise, (ok, I lied a little ).

There is no reason why the “average” admin can’t accomplish what Gary did. It starts with looking at each part of the network and each project you take on from a security standpoint. Simple questions like “Can I log these transactions somewhere?”, “How could this be exploited?”, “Can I know who connects to this and when”, at the start and both implementing and documenting everything you can goes a long way to helping you when the “proverbial” hits the fan. Computers are cheap and drive space is cheaper. An old PC with a nice big 80 Gig drive is “chump change” today. Couple that with a CD writer to archive the logs to at less than $0.50 per CD and your ability to log your system properly is acheived in less than $2-300. What’s the problem?

“Google it” is a mantra amongst security aware admins, and it’s a good one. I would propose another mantra. “If it communicates, log it”. I wouldn’t want to be blind in my real life, why would I want to be blind in my cyber life?

Finally, (”Thank god” yells the crowd), think about this beforehand. You have time during your commute, over lunch or even in the shower. When things start coming together use any other spare time to create your “procedure”. It might not be perfect but it will make the stress managable and it might, with good logs, help you do what you need to do…. Which is better than running around like a headless chicken while the sky falls……

Do not use, republish, in whole or in part, without the consent of the Author. TheTAZZone policy is that Authors retain the rights to the work they submit and/or post…we do not sell, publish, transmit, or have the right to give permission for such…TheTAZZone merely retains the right to use, retain, and publish submitted work within it’s Network

Code: Select all

Tiger Shark from Antionline has kindly given his permission for his tutorial to be hosted at The Taz.

You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=259017

Enjoy


Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.

This is a story. It’s fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the “clean-up” and mitigation of the breach.

Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.

—————————————————————————————————————–

“It wasn’t a voice I recognized” Mike said to Dirk who was standing nervously in the doorway.
“You mean you don’t think it’s an employee then?”
“I know practically everyonehere and the voice just doesn’t fit, no-one has an accent like that here”
“Well, then it does point to a “call home” program then. I looked at where the email came from and it was Amy’s.”
“Amy? She’s been here forever.”
“Well like you just said it doesn’t seem like an inside job and Amy doesn’t sound like a man either”
“Oh, yes, you’re right. So he want’s ten thousand dollars or he starts using the accounts.”
“Well we can’t close all the accounts and issue new one’s. That would tell everyone we were compromised”
“Can you find this ‘Al’? I saw a program on TV once about how a spy was caught. Can we do that?”
“I don’t think so. I need to look at Amy’s computer but finding the information I would need is going to be difficult. I could really use some log files but I took a quick look around and where there are any logs they overwrite themselves when they fill up which sometimes seems to be overnight on some machines.” [16]
“So you’re saying that without logs you may never know where all this came from?”
“Basically, yes. They are one of those things that always seem to get the low priority. I’m guessing the previous contractors had better things to do than to bother with them.”
“So we don’t have a lot of choice then? We are going to have to pay” Mike said glumly.
“Give me some time, maybe I’ll get lucky. Maybe Al is a little sloppy. When does he want the money?”
“I don’t know, he said he’d contact me soon”
“Let me look around. Call me when you know more, ok?”
“OK, I have some calls to make, get back to me if you find anything”
“Will do.”

Gary asked Bill Steel to call an emergency board meeting for 2:00pm today. He needed the ‘ok’ from the board to move to “investigative” pace. He had explained to the IRT that while the probability was high that he would find the outside locations that the attack had been initiated from the chances are that they were zombies being used by Al to mask his true location and identity. The recommendation would be that the company pay Al and move on. It irked Gary a little but he knew that the chances of successfully finding and being able to prosecute were minimal and that calling in the FBI would affect the company badly since the theft would become public once they were called in. [17]

Arriving back at his office Gary decided to take a look and see who’s computer in Cincinnatti was carrying out the scan. A quick check of his DHCP logs indicated that on the day and time of the scan the computer belonged to the manager of the Cincinnatti office, Dan Ereg. “Interesting.” thought Gary not doubting that Dan wasn’t Al but beginning to wonder how pervasive this attack was. “I do hope he hasn’t been hopping from box to box for weeks,” he thought, “that’s going to be a real pain not to mention the fact that I’m going to have to explain why I didn’t notice it”. [18]

Dirk sat forlornly in front of Amy’s computer. He’d opened the event logs to find nothing. “This really sucks,” he thought, “those contractors are useless…. they didn’t even turn on auditing. Jesus, all they had to do was set the domain policy…. Less profit in doing it right I guess”. Amy’s voice brought him back from his angry thoughts.

“You don’t look very pleased” she said, “Is it something I’ve done?”
“No, I don’t think so. I’m peeved at the old contractors. I’m trying to find something that should be easy had they done their job correctly…. But they didn’t”.
“What are you looking for, maybe I can help”
Dirk laughed, “I don’t think you could help me, I don’t know what it is I’m looking for myself”.
“That doesn’t make much sense,” Amy replied, “How do you even know you should be looking for something then?”
“Ahhh… It’s a long story. Maybe I can tell you some other time, when I get this all cleared up”.
“Oh please, you’ve certainly got my curiosity up” she smiled. The double meaning went right over Dirk’s head…….

Dirk went back to looking around her computer. He was lost and he knew it. “This is bloody silly,” he thought, “Thirty thousand files or more and no idea where to start. If I’m not careful it’s going to look like Amy did it and I can’t believe that.” He wandered aimlessly around Amy’s computer for another twenty five minutes and was ready to give up when he opened the task manager. Looking slowly down the list it struck him as odd that mstask.exe was running. “What’s this?” he thought, “The scheduler is running”. he opened the task scheduler and there was a single entry named “Daily Backup” set to start at 6:00pm daily. “Hmmm, it ran last night at 6:00 and it is due to run again at 6:00. What’s it doing?”. He quickly checked the properties. What he found made him sit up suddenly. The task was scheduled to run a program from the system32 folder called “msbackup.exe”. “Got it.” he thought, “Microsoft’s backup program is “ntbackup.exe”. This must be a trojan. There’s an IP address amongst all the switches. I wonder where that is?”. He quickly made a note of the address and returned to his own workstation. He “pinged” the IP address and got a response. “Cool,” he thought, “It’s up. Now what?” [19]

Gary looked up as the technician knocked on his door carrying a copy of Mike Panoff’s imaged drive and a manifest detailing what had occurred and where the other drives were currently located. He thanked the technician, filed the manifest after checking it to make sure things were in order and placed the drive in his machine. He went straight to the scheduled task which he found to be called “Daily Backup”. “No reason for that whatsoever” he thought as he looked to see what the executable was. “Now that’s an interesting command line”, he thought. “msbackup -l 80 -e cmd.exe….. that looks all too like Netcat… running as system too because of the scheduler, ouch, nice one.” He made the appropriate notes and moved to the event logs. There he found new processes being started just after the portscan had taken place. “VNC was run not long before the access to the financial database.”, he thought, “OK, so he’s even making himself a desktop to work from. Lets take a look at the VNC executable”. He opened Explorer and went to the path shown in the event log. Nothing. “Ok, let’s try the recycle bin?” he thought as he switched to it and found it immediately. “Great, the deleted time was not long after the data was stolen.” He further noted that there were a couple instances of the VNC executable over the previous few days. “Hmm… He downloads a new version each time. I hope he only uses Dan’s machine as his first stop internally”. He sat back to think for a minute. “Why was he portscanning for port 80? He has to know he has Netcat on Mike’s box….”. It took a full three minutes before the possible answer came to him. “He can’t guarantee which computers will be left on after work….. Shit, he has several machines….. I bet there are 18 machines total….. I need that portscan log again….” [20] He called the network admin and asked that all 18 machines on the list be checked for the scheduled job and was surprised when, just 45 minutes later, the admin was standing at his door with log files of his technicians activity on each machine as Gary had requested.

“You were spot on.”, he said, “Every box has the same scheduled job. Busy little beaver that hacker friend of yours eh?”
“Jeez…. He wanted to guarantee access didn’t he?”
“So what do you want me to do. I can’t pull all the boxes, I don’t have replacements.”
“Yeah, I know…. tough one…. The board meeting is in 5 minutes so I can’t say right now. We’ve recommended no action so we can move faster but the board has to ratify that. Until then I’m supposed to work as if we were going to court with this stuff. Then again, if we have good evidence on Mike’s box and we pick up the box in Cincy, we should be good in a court, anything else we have should just be additional nails in the coffin so just documenting the other machines should be good. It’s something I’d have to check with the legal beagles if the board wants us to go after him but we should know in an hour or so. Let’s just wait till then.”
“Ok, he’s a pretty bad hacker though isn’t he? He’s leaving clues all over the network.”
“Yeah, I would have expected better, even the log files were intact. Hopefully he won’t be any more careful with the box or boxes he uses to enter the network”

Dirk was now uncertain where to go from here. He’d found the attacker’s computer in the internet, it was up and running and he was somehow controlling Amy’s machine to do whatever he was doing to get to the financial data. “I need a plan,” he thought “this can’t be the first time something like this has happened, someone has to have seen this before…..” Finally, the “light bulb” came on…. “Google” he thought, “I’m a genius….”. He scrambled over to his workstation, opened IE and went to Google. He typed “incident response methodology” and clicked “search”. “Oh Hell. 201,000 hits…..” He added the word “windows” and re-searched. “That’s better” he thought, “cut that down to nearly 42,000 hits. Shit, I’m going to be reading all night….” Despite the seemingly overwhelming task Dirk felt a kind of warmth. He wasn’t alone, he wasn’t the first victim and there was a lot of information out there about how he should proceed. “Damn, I probably should have done this earlier.”, he thought. [21]

“You have got to be shitting me”, Gary said, “Do they know what that will do?”
“Look Gary, that’s the decision of the board. They want you to go as far as possible and then they will call the FBI.” Bill said.
“You explained that our chances of finding this asshole is close to zero didn’t you?”
“Yeah, but the feeling is that if we give the ten grand he’ll be back anyway. Eventually he’ll drop the information somewhere and then we’re screwed regardless. They feel it is better to face the issue head on, tell the investors about the leak and do everything to minimize the damage. The PR people are already working on the issue. Sorry, but you get to stay on the slow track”
“Hell, I have eighteen boxes I have to drag off the network already…. Should we just close up shop now?”
“Eighteen, how?”
“It’ll take me too long to explain but he has eighteen boxes on this network alone he can connect to and control, never mind the other offices.”
“Nice, very nice….. Can we handle that?”
“I dunno, listen, I have a thought, what do you think? I’ll just leave the boxes up, I already took down Mike’s but that should be ok, he expects boxes to be turned off sometimes. If I leave the other boxes alone and let things happen as they will can we still hold up in a court? I want to just put a monitor on all of them and see if he comes back. If we maintain a proper evidence trail on the boxes we have can we allow the other boxes to be tainted and still win on the off chance we ever get this shit into a court?”
“If your solid on the evidence trail of the boxes you have right now and can show his activity from your monitors I’ll argue it with any defense attorney but I need a rock solid place to start and I’ll need good data to back up our decision. Can you provide that for me?”
“I think so, but you’re the legal beagle…. Do you trust me is the question?”
“Er…. Oh, to hell with it…. Go for it… Make sure you are letter perfect on the procedure with everything you do. I need everything documented and I need you to be able to show reasoning. You do that and I’ll back you and argue it in any court if we can get it there”
“Thanks, I appreciate that. I’ll set up the monitors now, the jobs will kick off in, um, one hour, twenty three minutes…. I’ll be ready”
“Good, and good luck… to us all”
“Yeah, we need some, bye.” Gary said as he put the phone down. [22]

“Now this really sucks” he thought. “I really don’t like this…. I’d rather close all this down now…..”

—————————————————————————————————–
The Notes:-
[16] Dirk is beginning to realize that the log files on a network are his eyes. When log files don’t exist or overwrite themselves when “full” he is blind. All manner of things could have gone on throughout his network minutes before the log began overwriting itself and he would have no way of knowing it.

[17] No matter how good a sleuth you think you are you must realize that once the trail leaves your network it also leaves your logging systems. Once this has occurred the trail will most probably “dead end” fairly quickly. Your task is also to operate in the best interests of your company rather than follow your desire for revenge against your “violator”. The IRT’s recommendation to the board must be realistic and take into account the various aspects of the whole and their repercussions.

[18] Whether you have the best logging systems in the world or not you can’t expect to find every little thing. Even on relatively small networks the traffic volume can be huge and diverse. A good attacker will try to utilize “normal” traffic patterns to mask his malicious activity. Accept the fact that network traffic is extremely complex and unless you packet capture everything on the network you are going to have holes in your logs that may allow malicious traffic to slip by. Do your job, if you have thought about this situation previously and tried to make sure that you log the “right” things you should still be able to piece a trail together. It may not be complete but it will tell you were ‘Al’ has been which makes cleanup a lot easier.

[19] Bad move Dirk. He might as well have called Al and told him he had found one of his tools. You shouldn’t make any direct contact with remote machines at any time during the investigation until you have collected all the information you can. Once you have that information use a dial up connection to another ISP or go and use your home computer to look at the remote machine. You have to be very careful wherever you take a “peek” from. Al is in the process of committing a crime. One that could cost your company a lot of money and more importantly put him in jail with a nice cell-mate called Bubba. He’s going to be more than a little suspicious about probes against his machine(s). You really don’t want to force his hand and have him release the information for example when you were trying to keep the compromise quiet.

[20] While there isn’t always a good reason for things appearing in logs there are often sane ones. Gary nearly bypassed the reason for the portscan and may have become sidetracked by more “exciting” tasks. It’s important to look at each event that is relevant to the attack and try to determine why it occurred. Gary got lucky, the thought occurred to him after the fact and he didn’t dismiss the question. Had he done so he wouldn’t have found all the holes in his network. It helps to try placing yourself in the position of the attacker and work backwards. Why would I portscan this subnet from Cincinnatti trying to find my own Netcat? That question may come up with an answer much quicker than “Why would he be scanning me internally”? The difference is subtle but if you can place yourself in a position where the attackers “problems” become your’s you may speed up the process.

[21] No matter what is happening. No matter how stressful it might be, you aren’t alone and you aren’t the first to have been cracked and have the company’s data and it’s reputation on the line. No matter how well prepared you are you will most probably be ill-prepared for something that turns up. Let’s face it, few of us are full time network security analysts with daily excursions into the world of forensic investigation. Most of us are network admins who rotate ten different hats every day. But that’s ok, Google will help. It’s a mantra repeated hourly in the computer security world, “If in doubt, Google it” or “Google is your friend”. Learn it and live it. Again, it is better to spend time finding out what you _need_ to know to be successful than to ruin your chances of ever being successful.

[22] Unless it’s your company and your money things aren’t always going to go your way. You might understand how low the chances of finding the attacker are going to be but it is the board’s decision as to how they run the company and what may serve it’s interests best. You give the best advice you can, you are clear about your abilities and your inabilities and you let them make their decision…. It’s their business and that’s why they get the “big bucks”. It’s important that they know your weaknesses as well as your strengths. It may seem foreign to be telling your bosses that you “suck” at something but it is information they need to know. Suggest that they hire a contractor to help you in your areas of deficiency. The important thing is that you reach the truth. Unless they are utter idiots they will appreciate your proactive approach to _their_ problem and your understanding of your own deficiencies. It might sound like “falling on your sword” for the good of the company but if they don’t understand and appreciate your committment I can assure you there are better places for your talents.

Do not use, republish, in whole or in part, without the consent of the Author. TheTAZZone policy is that Authors retain the rights to the work they submit and/or post…we do not sell, publish, transmit, or have the right to give permission for such…TheTAZZone merely retains the right to use, retain, and publish submitted work within it’s Network

Code: Select all

Tiger Shark from Antionline has kindly given his permission for his tutorial to be hosted at The Taz.

You can find the original post here:
http://www.antionline.com/showthread.php?s=&threadid=258977

Enjoy


I really apologize for the long period between “episodes”. It was due to two factors:-

1. Radisson Hotels and Orlando International Airport in Florida are somewhat retarded. No wireless at the airport, and a choice between internet over the TV or 89 cents a minute to dialup the internet from the laptop. Neither of which was accepable or useful…..

2. My sweetie and my friends kept me sufficiently busy to make a quick wardrive impossible. When I wrote my note last night I was at a friends house for a BBQ, was a tad drunk and was being yelled at by my other sweetie, (our hostess), for working on the computer when I only see her once a year……

So…. To Part II

Title: A Tale of Two Log Files (Part II)
Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.

This is a story. It’s fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator in the event of a compromise, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the “clean-up” and mitigation of the breach. It’s a lot to do with the mindset and approach whether you are prepared or not and that is what I will try to show during the story.

Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.

This one was written with no technical reference available…. on a plane….. just ignore the inaccuracies…..

————————————————————————————————–

Following his procedure Gary had carefully carried out the remote portion of his investigation of Mike Panoff’s computer and was somewhat surprised to see that everything was as it should be. He called network administration and requested a new computer imaged for Mike Panoff’s department be brought to Mike’s office, picked up his forensic CD carrying his “toolkit” and made his way to Mike’s office.

“What’s up then Gary”, Mike said as he entered.
“Ahhh… Not much… There’s nothing wrong with your computer from my point of view but I noticed the CPU fan seems to be failing. I’ve ordered you a new box and it’ll be here in the hour. I’m doing the netadmins a favor and running the diagnostics for them”
“But what was it that made you look at it in the first place?”
“Oh, that….” Gary laughed, “Dumbass here misread a log file and transposed a couple of numbers. Those logs turn you cross-eyed after the first ten minutes” [8]

Gary and Mike carried on chatting as Gary mapped a network drive and began running through his repertoire of tricks sending the output away to the newly mapped drive. He didn’t have to look to his procedure book to determine what he should be doing next which may have raised Mike’s suspicion’s because he had built the CD from the procedure manual and had practiced with it. He had batch files written for the more complex switch requirements of some tools so that he wouldn’t have to remember the commands or refer to a manual for just this situation. All he needed to know was that he needed to run every tool on the CD, with the batch file if it was available and the location to send the resulting data. [9]

An hour and fourty seven minutes after leaving, Dirk was again parking his car in TFCU’s parking lot. His spirits were up from the initial low of the day and he had promised himself to have a “cold one” for Fyodor next time he had the chance. The NMap scans he ran against his sixteen IP public subnet had shown that there was no direct exposure to the public network. Small victory, but a victory all the same. He dropped into the CEO’s office on the way back to his own.

“Well, I have some news…. It wasn’t directly from the outside. I can’t find a way in from the outside”
“Er… But that’s not good news then. Doesn’t that mean that “Al” is an employee?”
“It could, but that is yet to be seen”
“I’m not sure I understand…. If it isn’t from outside then it must be from inside. Who’s computer did the email come from?”
“I don’t know right now, I haven’t checked. But I’m telling you that right now we don’t want to be accusing anyone of anything”.
“I’d like to know who owns that computer…. and I’d like to know pretty soon”.
“Mike, listen to me. There are a lot of ways that someone within the network messed up a machine without knowing it and now someone outside controls their computer.” [10]
“You just told me that there was no access from the outside, so that’s not possible”
“Mike, it is…. It’s like…. er… ET calling home…. I don’t remember the proper term but I read about it a few months ago. It’s a way around the firewalls” [11]
“So you are saying that even though no-one can get in from outside, the firewall we payed that contractor almost $3000 for a few years back is useless?”
“No, no… not at all. I have to let some traffic flow freely from the inside to the out or you won’t get your email, other staff won’t be able to use the web etc. etc.. So if Al got a program inside here that calls home through the… um, web interface that I have to have open then there isn’t a damned thing I can do to stop it.” [12]

Gary leant back in his chair as he addressed the meeting. The entire team was assembled and he had outlined the initial evidence that indicated a compromise had taken place.

“I just completed the data collection from a user’s computer that may have been used to steal the data. The computer has been switched off and removed from the user’s office and he has been given a replacement. I took a quick look at the data I collected and there is one thing that jumps out at me that seems a little odd but I need to get the drive imaged and secured before I can look deeper into that.”
“Can you be more clear” said Bill Steel, the legal representative on the team, “I mean do we have a suspect or are there other possibilities”
“Well, at this point I’m not going to point a finger anywhere. The odd thing on that box is a scheduled backup for 6:00pm nightly. It’s odd because by policy all user data is forced up to the file servers to keep it safe so there is no reason to run backups on the client workstations. However, until I can access a copy of the drive I can’t determine what is really there and why.

The meeting continued as the parties moved, step by step, through the series of questions they need to have answers to so that they can determine the course of action the company was going to take in this instance. The sticking point was “Al”. Having not had subsequent contact with him the team had no way of knowing what he was going to request. Clearly it was money but to a banking institution the amount would be of great interest. The meeting adjourned with no initial recommendation which would be postponed until after “Al” had made his demands clear. This bothered Gary somewhat in that until the recommendations were formulated and accepted by the board he had to treat everything as if it would have to be presented in a court of law and this would slow him down. He’d made that point at the meeting to ensure everyone was aware. [13]

Dirk sat staring blankly at his monitor. “Oh dear…. Why did it have to be her box, why couldn’t it have been that moron in Sales? I wouldn’t mind chasing him down and getting his ass nailed” he thought. After leaving Mike’s office he had decided to start by looking at the computer that sent the email last night. Having checked his records the IP and DNS name of the machine indicated it was Amy’s computer. [14] His mind gyrated as he tried to determine how to approach her and what to do when he got there when suddenly he thought, “Hah, it can’t be Amy….. Yes!…. She has no access the the database….. perfect…and I can prove it…. The server’s log files will prove that.” He scoured the log files for the previous week searching for anything that would point to Amy accessing the server. He was disappointed until he saw the time of the access, 10:27pm six days ago. There wasn’t a successful login but there had been two attempts. “Two tries” he thought, “Someone didn’t want to trip the automatic lockout. It couldn’t have been her, she would have used her password”. Poking around a little more he came to the transaction logs for that date. Scrolling down through the endless text he noticed some odd entries. “What on earth is that?” he thought, “Never seen entries like that…”

SELECT username FROM users WHERE MID$(username, 3) = “m” AND 1 = 1;

There were lots of them, rotating the numbers and the letters in the “where” statement. After looking at the entries for a while he thought he could see what was going on. “Someone is doing something to the administrator name in the user tables”, he thought, “But what’s the ‘AND 1=1′ all about?” He was familiar with basic SQL queries but his practical knowledge was limited. Looking further down through the log he could see where the queries that extracted account information from the customer tables. “Well, for right now I can see where the leak took place but where the queries originated from is anyone’s guess. I’ll make backups of these logs and then see if I can find where the queries came from.” he thought. [15]

Gary moved across to his Secured Logging Systems analysis console while the image of Mike Panoff’s drive was being completed, documented and secured. “No point in sitting on my hands”, he thought, “If I’m lucky the internal IDS logs might show something of interest on Mike’s box”. He filtered the output to show only alerts from the internal sensors and started looking through the logs. While it isn’t unusual for workstations to trigger portscan alerts one caught Gary’s eye.

Portscan from 192.168.70.153 Ended: Time: 12 seconds, Hosts: 18, TCP: 18, UDP: 0

“What’s the Cincinnatti office doing kicking off a portscan? It must have only just exceeded the threshold”, he thought as he opened the portscan log itself. “The time is right and if Mike’s box is one of the target boxes then things may be coming together”. He looked down to the appropriate time and date and found the offending IP address. There it was, Mike’s IP address had been scanned from Cincinnatti about 30 minutes before the login to the server took place. “Something isn’t right… ” he thought, “the scan is against port 80, Mike’s box didn’t have 80 open. I’m going to need that image to follow this track. First let’s see who our new “potential perp from Cincy’ might be, maybe we have ourselves a winner.” he thought reaching for the phone. His hand didn’t get to it before it rang. He picked it up expecting to hear that the techs were done with the drive imaging procedure and that he could get it back.

“Hello, Gary speaking”
“Tell your board that the price is ten thousand dollars. I’m not a greedy lad and ten grand isn’t much to your bosses, they could probably each pay it from their own pockets and not miss it”.

Gary grabbed for a pencil after the initial shock of hearing “Al’s” voice again. He needed to get the converstion down verbatim if he could but he already knew that he wouldn’t be able to do that. “What I can scribble down is what they are going to have to get” he thought as he checked his watch and noted down the time.

“Ten grand is a nice chunk of change, I dunno what they are going to say about that” Gary replied
“Bullshit, and you know it. The information I have here is worth way more than ten grand…. Tell them to be smart…. and keep your eye on your email”…

The click of the phone found Gary still writing frantically to try to document the converstion precisely. He’d done a pretty good job, “a word here a word there” he thought as he finished up and reached for the phone again. “We need another IRT meeting, and fast” he thought as he began dialing familiar numbers

—————————————————————————————————–
The Notes:-

[8] Here’s Gary lying again, but it’s all in a good cause. He’s come up with a story for why the box needs replacement and why it came to his attention in the first place. They don’t have to even be good stories for most users because they trust that you know what you are doing. Admitting to making a mistake goes a long way to having people believe you. It doesn’t matter that you didn’t make a mistake, the user is likely to empathize and accept your story more readily.

[9] This is one of the great benefits of being prepared. Gary doesn’t have to concentrate so hard on what he is doing to the exclusion of all else. His tools are all there, organized, easy to use and they follow the policy laid down. If the policy requires that certain tools be run in a certain order place them in folders on the CD called “1st”, “2nd” etc. The more you can do beforehand to make your task easy in the real event then the less stressed and more effective you will be. A further side benefit is demonstrated in the story, Gary can go about his difficult task while making it seem easy, stress free and routine without raising people’s suspicions. Furthermore, he can complete any documentation of the tools run and in what order from the time/date stamps on the files he is creating.

[10] Kudos again to Dirk. His boss is hell bent on finding a culprit quickly. He’s made a decision, based on a lack of information, and he now wants action. Dirk has, quite rightly, pointed out that there are alternatives that are equally probable and that rash action should not be taken. He also didn’t accuse anyone of doing something maliciously, he used the term “messed up” that implys an accident rather than using a more purjorative term such as “downloaded something” which implies a more deliberate act. This helps to keep the stress level of the major stakeholder’s down a little and can keep them from interfering in the wrong way.

[11] You don’t have to be a cracker to be effective in security. But you do need to keep up with what a cracker can and can’t do. You don’t even need to remember the details, just that it can be done. Without that knowledge it would have been easy for Dirk to also conclude that this is an inside job. Keeping up with the knowledge means you don’t have to discover things for yourself. It also allows you to be creative in your thinking, as a cracker would, in the ways these weaknesses can be exploited.

[12] Dirk thought about how he would phrase technical details so as not to complicate the issue with jargon unless it is absolutely necessary, (which is usually only at the conclusion of an investigation). This just isn’t the time to be throwing around port numbers and protocol names to stressed executives that have no idea what you are talking about. The odds are they will ask for clarification which costs you more time and there is a high probability that they will inadvertently misrepresent what you said to others thus confusing the issue and starting the rumor mill turning.

[13] Gary is on top of things as usual. It is critically important that the IRT understand that the investigation must proceed from the beginning at a “litigeous” pace, meaning that the technicians are expected to do everything as if it will be presented in a court of law. This is much slower than moving at an “investigative” pace which lacks much of the documentation and evidence preservation required by a court of law. It is also of critical importance to ensure that the board understands that when you will move from the higher requirement to the lower the evidence will be tainted. It must be stressed and understood by both the IRT and the board of directors that the two methodologies are exclusive and once the litigious methodology is departed from there is no going back with any hope of a successful prosecution.

[14] Regardless of the fact that there are people you like, people you dislike, people you trust and people you don’t there is one single rule in an investigation that you must follow. You distrust and dislike them all equally, it’s as simple as that. If you don’t you will allow preconceptions to cloud the process and possibly, unwittingly, allow them to move you away from the “truth”.

[15] Dirk’s lack of preparation and knowledge is beginning to show through. He’s “playing” with a log file that may contain evidence but he isn’t documenting it and he didn’t make copies before he looked at them. If Mike choses to try to prosecute the perpetrator in the future the chances are high that he will never get the case to court. Now he’s found his evidence he is making the appropriate copies, but this is too late. The copies should be made first, preferably to “write once” media such as CD-R if a litigious course is to be taken and then they should be searched and manipulated from the read only copy of the backup.