Network troubleshooting, what would you do?

[Squeetard]

I got called out to a Sawmill last Thursday, the entire network was down. Nothing worked everything was active, you couldn't even ping the computer plugged into the same switch as yours. The mill has three buildings, head office, sawmill and planer. They are linked by fibre and each building has a few 24 port switches in them.

Knowing the above, how would you go about trying to find the problem?

This is not a trick question. use common sense and tell me.

[BigDogg]

Originally Posted by Squeetard

I got called out to a Sawmill last Thursday, the entire network was down. Nothing worked everything was active, you couldn't even ping the computer plugged into the same switch as yours. The mill has three buildings, head office, sawmill and planer. They are linked by fibre and each building has a few 24 port switches in them.

Knowing the above, how would you go about trying to find the problem?

This is not a trick question. use common sense and tell me.

Could it be possible DNS issues. Seems very very weird that two computers on the same switch cannot ping each other. I would try hardcoding the IP information to see if that allows them to communicate again. If that doesn't work try rebooting everything and see where you can get. If that doesn't work ensure that each switch works and then work backwards :P Seems like an issue with the core switch ?

[sswilson]

Remove the last windows auto-update that came through overnight? ;)

I'm not real big on the specifics of how a switch works, but I'm thinking that perhaps they don't do any DHCP themselves. Would you have to go back to the main device that's providing DHCP to the interconnected switches?

[Perineum]

Originally Posted by Squeetard

I got called out to a Sawmill last Thursday, the entire network was down. Nothing worked everything was active, you couldn't even ping the computer plugged into the same switch as yours. The mill has three buildings, head office, sawmill and planer. They are linked by fibre and each building has a few 24 port switches in them.

Knowing the above, how would you go about trying to find the problem?

This is not a trick question. use common sense and tell me.

Well...

Is everything set up with static IPs or is it normally run DHCP? Do the machines have 169.x.x.x IP addresses?

Assuming it's static IP for each machine and you still can't a ping from any computer within one of the 3 buildings I would make sure of the basics first. I would make sure the switch and their connections were working/connected/powered on. I may unplug and reboot with switches without the uplink just to see how it acts. If they still weren't connectable by IP it would start getting interesting.

However, I have a feeling it will be something further along that where I am going. Also, I'm going to be asking a lot of questions about the network that you would already know and take for granted.

[Squeetard]

Originally Posted by Perineum

I may unplug and reboot with switches without the uplink just to see how it acts.

WINNER!

To me this is just plain common sense. If nothing is working then you must start by isolating stuff. First thing I did was remove the fibre link between buildings. Found that everything but the stuff in the planer came back to life. Then I started unplugging wires at the planer and found the trouble. Total time 25 minutes.

Now here is the kicker. I got called out after they were already down for 6 hours, at $35,000 an hour in downtime cost. They had called in six network guys from the company, had the and lord knows what they were doing for those 6 hours. Now there is shit to pay, CEO's jetting in all WTF? And I am hanging these buggers out to dry. If thier pitifull attempts at trying to cover their asses get any funnier I will post them here.

[Perineum]

Originally Posted by Squeetard

WINNER!

To me this is just plain common sense.

.....They had called in six network guys from the company....

Interesting.

So these 6 guys have jobs and I can't get a job because I failed a "computer" (MS OFFICE) exam with Telus for a part time position.

FML.

[chrisk]

Squeets is in BC like Perineum. Hook this guy up for an interview!!!

[Squeetard]

I'll let you know when these dudes get fired Perenium.

[Perineum]

Haha, thanks but I really can't relocate..... I need to be near my son. If that makes me a sissy then just pretend I said "parole officer" instead.

[Squeetard]

We'll get your son a job too. So what if he's like 6, he'll still do a better job than these clowns.