Friday, September 30, 2011

CCNP - Job Done.

Yesterday I passed the TSHOOT exam and thus completed my CCNP. I had a tense afternoon as I eagerly waited for my certification tracking account to sync to with Pearson Vue but late last night the e-mail came through from Cisco Training telling me I had achieved the CCNP and I could claim my logos.

My experience sitting the TSHOOT exam was a much improved one from the previous test centre I used to go to. I was advised that my usual company had actually gone out of business. So here I was at a strange test centre hoping to claim my prize.

I'm not going to test the NDA but I will say that the experience of the exam was like none I've ever encountered. It's on record that you can dip in and out of the tickets as you see fit. If you get stuck on one question you can select Abort and check the config in another ticket so see if you're right. I found I needed to do this on 2 occasions to confirm my line of thinking and but it felt like I was cheating some how. In all other Cisco exams once you move on to the next question you can't go back.

If I'm honest I'd have to say they could probably do that on this exam. You have to answer all the questions any way and I submitted them in order so that I could manage my time correctly.

In previous posts I've mentioned approaches to tackle the exam, Kevin Wallace recommended the Abort method above, one other method I used was to let the question guide me. On more that one question I had narrowed the problem to a particular device (so that's Q1) and could tell that a particular technology was the issue (so that's Q2 sorted) but I struggled to pin point the cause. I simply reviewed the answers available to me in Q3. Once you eliminate the ridiculous answers you're left with 2 or 3 plausible answers that you can work with.

My final point is about time management. I found this exam very enjoyable and once you pinpoint the cause a problem you rattle through the 3 part trouble ticket quickly. If you know your show commands and can comfortably recall the correct command to display the most helpful information then I genuinely think you shouldn't worry about the time.

Right. Time for a rest then I'm going to put some serious consideration as to whether the CCIE is worth my effort...

Tuesday, September 27, 2011

CCNP - TSHOOT - TSHOOT in the real world...Part2

The second trouble ticket that I was able to demonstrate my TSHOOT knowledge was the following:

Trouble Ticket #2 - the customer reported that hosts at SiteC were not getting a DHCP allocation from the server at SiteD.

The customer reported that when they performed a traceroute from the LAN at SiteC to on the LAN at SiteD the trace stopped. The network looked like this:

SiteC --> R1 at SiteC --> 'the core network' at R2 --> R3 at SiteD --> DHCP Server on SiteD LAN

For this one I took a look at our Nagios Monitoring and could see that there were no issues reported on R1, R2, or R3. As such I jumped on to R1 at SiteC and performed a traceroute to The trace was successful! Yet from the LAN at SiteC the trace failed?

Based on this information I formed my hypothesis. I asked the customer whether there could be an access-list/firewall between R3 and the DHCP server that could be blocking traffic from the LAN at SiteC? The customer investigated their equipment and found that an earlier change to an access-list on their equipment at SiteD was now stopping DHCP requests from SiteC. They corrected their configuration and the issue was resolved.

Trouble Ticket #2 - Closed

With this ticket because our network monitoring was reporting no issues with R1, R2, R3, I took the 'Shoot from hip' approach and looked to eliminate all L3 issues in one go. The trace route worked indicating that the server was responding to ping and that all L3 connectivity between SiteC and SiteD was fine. I could have tested the L1/L2 physical connectivity on each router in turn but our monitoring would have pick this up  and therefore I was confident the physical connectivity was fine. Once L3 connectivity had been proven on the equipment I have admin access on I could formulate my hypothesis and put it to the customer to test.

CCNP - TSHOOT - TSHOOT in the real world...

When studying for any certification exam it's very easy to end up concentrating on the goal of passing the exam instead of the actual goal which should be gaining knowledge and achieving a certain standard of expertise that can be practically utilised in the real world. The exam simply confirms that you have a certain level of knowledge. The point is to enable you to use it in the real world.

So here I am, studying for my TSHOOT exam, which will complete my CCNP, and in the week before the exam comes 2 real world examples of how studying for the TSHOOT exam has helped me investigate and resolve the issue in hand. The next 2 posts will outline the problem, methodology used and how the TSHOOT exam materials directly related to resolving the problems.

Trouble Ticket #1 - The customer contacted us to state that SiteA no longer had access to the internet via it's default route. 

 Following the TSHOOT troubleshooting methodology I gathered as much relevant information as I could from the client. I then reviewed the information, eliminated the possible causes until I was down to a small   number of possible issues, I then formed my hypothesis, tested it and reviewed the outcome.

For this particular problem the customer stated that following the closure of one office and the migration to a new site access to the internet via it's default route was no longer possible. The customer supplied me their default route - and that it originally routed via the now closed office. I also had the new network diagram outlining the new office and the appropriate routes. The diagram was essentially as follows:

SiteA Firewall ---> R1at SiteA ---> 'The core network' at R2 ---> R3 at SiteB - (trunk link)--> SiteB L3 switch ---> SiteB's ASA firewall --> INTERNET

First job was to see where the routing began to fail. So on to R1 and check the interfaces were up (yes) and then I moved on to the routing table to check for a route for This was a static route pointing to our core network (for this article we'll call all of it R2). I performed a traceoute on R1 to and it failed at R2.

On R2 I did the same, checked that interfaces were up and then checked the routing, This is where the first issue occurred. The static route on R2 was still pointing to the closed office. So the change was made. Static route was now pointing to R3 at SiteB.

I performed a traceroute to the default route from R1 again and it still didn't work. This time the trace stopped at R3.

On to R3 the routing table revealed that there was no route for at all. After referring to the customer they advised me that the traffic had to go via a specific VLAN on the L3 switch so on R3 we needed to create VLAN10 and allowed VLAN10 on the trunk link to the L3 switch at SiteB. Finally the route for was set to go via the SVI for VLAN10.

With this created I tested the route to from R3. It still didn't work? With the information at hand I was able to formulate my hypothesis.

Was VLAN10 a) created on the L3 switch? and b) allowed on the trunk link?

Next I checked the state of the VLANs. VLAN10 was in place on R3, and was allowed on the trunk link to the L3 switch. The trunk link was up and protocol was up. Everything that I had admin access to was now properly configured and had been tested however the traceroute was still not returning. I put this to the customer who inspected their L3 Switch. They reported that VLAN10 had not been created after all on their equipment.

Once the customer applied their configuration on their equipment the route from SiteA to the ASA firewall on the LAN of SiteB worked. It was only after the issue was investigated that it became apparent the customer had neglected to account for the routing required for SiteA when they planned the closure of their old office. As a result none of the required configuration had been prepared before hand and SiteA had been cut off once the old office closed.

Trouble Ticket #1 - Closed.

As you can see on this occasion I was able to use a 'Divide and Conquer' trouble shooting approach. Starting L3 and then checking L1 and L2 when the traceroute failed. On R3 I was able to narrow the issue to L2 and formulate my hypothesis which was then tested.

Saturday, September 24, 2011

CCNP - TSHOOT - Troubleshooting the Routers...

Continuing my TSHOOT methodology, in this post I'm going to look at troubleshooting the network between R4, R3, R2,R1 and out to the web server at

Having not sat the TSHOOT exam the impression I'm getting is the Trouble Ticket (TT) will state something like Client1 cannot access [insert resource]. As such you need to identify the system the issue is on and the resolution.

Again I'll be referring to the actual  TSHOOT topology - here

Let's say Client 1 cannot access the web server at Based on my previous article, also assume Client 1 can ping at R4. In this post I will outline how I see myself troubleshooting the connectivity issue to the web server.

I'll want to prove connectivity to R1. So in turn I'll ping each interface on the way to R1 at If at any point the ping is not returned then log in to the router that you were trying to ping and begin troubleshooting.

1) Are the interface on the link UP/UP? check that one of them is not admin'd down:
#sh ip interfaces brief

2) With interfaces up, is the L2 Frame-Relay topology correct? the diagram shows that the routers are connected via point-to-point links therefore I'm expecting to see a config similar to:
R4(config)#int s0/0/0
R4(config-if)#encapsulation frame-relay point-to-point
R4(config-if)#no shutdown
R4(config)#int s0/0/0.34 point-to-point
R4(config-if)#ip address
R4(config-if)#frame-relay interface dlci 403

3) Assuming L2 is fine move on the routing protocols. Things to check here are:
i - is route redistribution functioning correctly on R4? check the routing table, check config
ii - Are the adjacencies up? Check the neighbor tables/topology tables.
iii -Are the correct networks being advertised to the neighbor? check the ospf network statements
iv -  is there any authentication in place? if so it must match at each end
v - is the routing table including the correct route as per the network diagram? If the route is not in the routing table it won't be advertised out to the neighbor.
vi - Is the routing config command 'passive-interface default' set? if it is then updates will be suppressed by default. You need to either remove the command using #no passive-interface default
Or you specifically list the interfaces you want to participate in dynamic routing using:
#passive-interface default
#no passive-interface f0/0
#no passive-interface f0/1
and so on...

4) Up to R1 I think that would cover off all the major connectivity issues you'd be likely to get. If you still have problems look closer at R1.

On R1 check you can get to the next-hop outside of the network. If you can the config on R1 should be sound. If you can't it could indicate any of the following.
i - Check over the interfaces as per previous points. Ensure they are UP/UP. if they aren't then troubleshoot accordingly.
ii - Check routing/BGP - is the BGP peering up? if not check the config, if the update source being used is a Loopback address then you need the update-source command set in the BGP config. Check the IP's of the Peers, check the remote-as command is correct
iii - Is there a default route? I'd expect there to be one. Simple question. Is it pointing to the correct outbound next-hop? not in towards the LAN?
iv - Are there any access-lists? if so, what are they permitting and what are they blocking? Could the implicit deny  be causing an issue here?
v- Is the default route being correctly redistributed? I'm expecting to see something like:
R1(config)#router ospf 1
R1(config)#default-information originate
vi - Is NAT in operation? check which interface is inside, outside, check the ip nat source statements. Check any NAT pools that my be configured? if a pool is not configured then I'm looking for the 'overload' option in the command :
#ip nat source list 10 interface s0/0/1 overload
With out Overload then port address translation is not working and you'd only be able to have 1 host using NAT at any one time.

Based on the exam topology I'd hope that by running through my structured check list from L1, L2 to L3 to L4 and above that the issue in hand would be revealed.

When I'm in the exam I'll be taking my cue from the information in the TT. As such I might not need to bother with L1/L2 checks if the remote peer is responding as expected.

My next article is going to look at the IPv6 topology and look at how I might be troubleshooting problems there.

Tuesday, September 20, 2011

CCNP - TSHOOT - Considering the Client...

In the TSHOOT exam the bulk of the trouble tickets will pretty much be Client1 cannot access [insert resource here], identify the problem device and present a solution.

The TSHOOT topology is freely available from the Cisco website - here - and so we can take a look and start to suggest possible problems that can be presented. On it we have BGP, OSPF, EIGRP and redistribution, NAT, DHCP, IPv6 Tunnels, OSPFv3 and RIPng redistribution, we've got first hop redundancy and etherchannel, plus OSPF over Frame-Relay. Some pretty tasty stuff.

So, where to start? Well, lets say that Client1 can't access the web server at As I'm looking at client issues assume that the config between the web server all the way in to the network to R4 is fine.

When I'm in the exam I'm going to be looking to ensure connectivity to Client1's default gateway and the DHCP server at R4 is sound. When I find that Client1 is unable to ping the default gateway or obtain a DHCP allocation here is what I'll be looking for:
1) Client1 IP Assignment - The L2/L3 topology diagram states that the address is obtained via DHCP (R4 at On Client1 do >ipconfig - if the address is move to ASW1, if the address is 169.X.X.X then DHCP is not working properly for some reason. In both cases move to ASW1

2) Interface VLAN membership - Client1 should be connected to int f1/0/1 - is it? if not then correct as appropriate. Interface f1/0/1 should be a member of VLAN 10 - does the VLAN exist? :
#sh vlan

Is int f1/0/1 a member of VLAN 10?:
#sh run int f1/0/1 OR #sh vlan

Does ASW1 have any port security?
#sh run int f1/0/1
if so what is configured here? Does any static MAC address match that of Client1? does the port limit number of MAC addresses per interface? Problem is here is if port security is incorrect. Otherwise move on

3) Is VLAN 10 allowed on the trunks?
Firstly prove int Po13 is an active Etherchannel and that interfaces f1/0/19-20 are members
#sh etherchannel summary - Trouble shoot accordingly if this is not the case.

Next is int Po13 a trunk link?
#sh interfaces trunk - if not then problem is here.

Next which VLAN's are allowed on the trunk links?
#sh interfaces po13 trunk - if VLAN10 is not here then you won't get to DSW1

Next, can you see any VLAN filtering in place? if so check that any permited ACL has the correct ip range otherwise traffic from will not be allowed to use VLAN10. Also check the vlan access-map config ensure that the traffic allowed in the ACL is forwarded not dropped.

Do step 3 on both ASW1 and DSW1. If one side is not configured correctly then connectivity will fail.
If everything looks good you should be able to at least ping VLAN10's dfg at on DSW1, move on.

4) Check routing on DSW1:
- DSW1 is an L3 switch that needs to send your DHCPDISCOVER to the DHCP server on a different network. As such ensure ip routing is enabled:
DSW1#sh run | begin ip routing
- Check L3 connectivity between DSW1 and R4 - Ping, check interfaces are UP/UP
DSW1#sh ip interface brief
- Check DSW1 int f1/0/1 is not admin'd down:

- If all looks good here check that the ip-helper command is applied to interface VLAN 10:
DSW1(config)#int vlan10
DSW1(config-if)#ip helper-address
(this will tell DSW1 to forward UDP broadcasts received on this interface)

5) You should now be at R4. If all of the above looks good then you should look above L2/L3 to DHCP or other factors.

6) DHCP configuration:
- Is DHCP running on R4? look for service dhcp in #sh run
- Are there any leases active? check - #sh ip dhcp bindings
Check the Hardware Address with that of Client1 to ensure the correct Client has an assignment. If you've discounted steps 1-4 I'd be surprised if this was the case.
- Check the DHCP address pool is correctly configured:
DSW1#sh run | begin ip dhcp
look for any kind of misconfig here be it incorrect subnet defined for the pool or an exclude address incorrect

7) At this point I'd expect to have uncovered an issue. If that's not case there's still some tactics I can try.
i) Kevin Wallace suggested Abort! - if you think you've narrowed the issue to a point but are unsure of the final solution, hit Abort, dive in to a different ticket and cross check the config. If the config is different the here is your fault. If it's the same then you can discount that area as the source of the problem and move on
ii) Let the question guide you - you've narrowed the issue to, lets say, DSW1 but are still unsure. Review the options available to you in part 2 and part 3 of the trouble ticket and use the information there to gleen if you are looking in the right place.

I think I've covered off the major issues that could be presented. I guess HSRP on DSW1 could be an issue but if DSW2 is the active router (#sh standby) then you should still be able to reach the DHCP server at

Please let me know if I've missed a point or am miss guided in any way. Thanks

CCNP - TSHOOT - EIGRP Troubleshooting Targets

For EIGRP issues look to the following:

- If Adjacencies are not forming check:
Timers match, AS number matches
In Frame-Relay topologies check that you have included the 'broadcast' option to allow the multicast packets that are sent to to be forwarded.
Check for any ACL's blocking EIGRP specifically or the subnets in general
Check that the router interfaces are up, you can ping the neighbour ( if it isn't up then troubleshoot the interface config as appropriate)

- If Static/Dynamic routes are not being redistributed in to EIGRP check:
Routes learned by EIGRP are in the Routing Table
Check the EIGRP topology table to see if the route was learned at all
Check the EIGRP config to ensure you are advertising the correct networks
Are there any Distribution Lists filtering your network?
For discontiguous networks check that no auto-summary is applied.

- For the Error 'Not on common Subnet' check:
IP addressing the interfaces that should be forming an adjacency.
Remember EIGRP packets are sourced from the Primary IP address on the interface. As such you can't use a Secondary IP to form the adjacency.

- If Load Balancing does not operate as expected check:
Ensure you have correctly used the Variance command in the Router process (not at the interface)

- Do you get 'Stuck in Active' (SIA) errors?
Check that the router has enough resources to respond to EIGRP updates (#sh processes cpu)
Check the bandwidth parameter on the interface is set correctly
Check #sh interfaces for interface resets, input/output errors.
If you do observe these issues look at hardware upgrades.

Monday, September 12, 2011

CCNP - TSHOOT - Maintenance Models

The main models used in assisting with the maintenance of a network are as follows:

  1. FCAPS - which is - Fault mgmt, Configuration mgmt, Accounting mgmt, Performance mgmt and Security mgmt. Defined by ISO (International Organisation for Standardisation).
  2. ITIL - the IT Infrastructure Library, which is a collection of best practices. When implementing this PLEASE PLEASE PLEASE remember it is Best Practice not THE Practice. I've worked at companies who have idiots Service Delivery Mangers who think that to be 'compliant' they need to implement all of the guidelines regardless of how big the firm is or how suitable the process is to their enterprise </rant>
  3. TMN - Telecommunications Management Network, the ITU-T's variant to FCAPS tailored to the telecommunications field.
  4. Cisco Lifecycle Services - You should be familiar with this from your ROUTE and SWITCH studies Chpt1. Covering the PPDIOO - Prepare, Plan, Design, Implement, Operate, and Optimize life cycle.