CCNP - ENTERPRISE - Subnet Mask vs Wildcard Mask

Something I find I struggle with is when to use a subnet mask and when to use a wildcard mask. This article is to simply set out the instances when you will use one or the other (not the actual steps to apply them).

Subnet Mask:

• When applying an IP to an interface
• Routing protocol summary addresses
• BGP
• PIX security appliance ACL's
• ASA security appliance ACL's
• When creating DHCP pools on a Switch or Router

Wildcard Mask:

• EIGRP network statements
• OSPF network statements
• VPN concentrator network lists (when setting the local and remote allowed networks)
• Router ACL's

This list is not exhaustive and I will add to it as I come across new instances where you use either a Subnet mask or Wildcard mask. 

CCNP - TSHOOT - TSHOOT in the real world...Part2

The second trouble ticket that I was able to demonstrate my TSHOOT knowledge was the following:

Trouble Ticket #2 - the customer reported that hosts at SiteC were not getting a DHCP allocation from the server at SiteD.

The customer reported that when they performed a traceroute from the LAN at SiteC to 192.168.10.10 on the LAN at SiteD the trace stopped. The network looked like this:

SiteC --> R1 at SiteC --> 'the core network' at R2 --> R3 at SiteD --> DHCP Server on SiteD LAN

For this one I took a look at our Nagios Monitoring and could see that there were no issues reported on R1, R2, or R3. As such I jumped on to R1 at SiteC and performed a traceroute to 192.168.10.10. The trace was successful! Yet from the LAN at SiteC the trace failed?

Based on this information I formed my hypothesis. I asked the customer whether there could be an access-list/firewall between R3 and the DHCP server that could be blocking traffic from the LAN at SiteC? The customer investigated their equipment and found that an earlier change to an access-list on their equipment at SiteD was now stopping DHCP requests from SiteC. They corrected their configuration and the issue was resolved.

Trouble Ticket #2 - Closed

With this ticket because our network monitoring was reporting no issues with R1, R2, R3, I took the 'Shoot from hip' approach and looked to eliminate all L3 issues in one go. The trace route worked indicating that the server was responding to ping and that all L3 connectivity between SiteC and SiteD was fine. I could have tested the L1/L2 physical connectivity on each router in turn but our monitoring would have pick this up  and therefore I was confident the physical connectivity was fine. Once L3 connectivity had been proven on the equipment I have admin access on I could formulate my hypothesis and put it to the customer to test.

CCNP - TSHOOT - TSHOOT in the real world...

When studying for any certification exam it's very easy to end up concentrating on the goal of passing the exam instead of the actual goal which should be gaining knowledge and achieving a certain standard of expertise that can be practically utilised in the real world. The exam simply confirms that you have a certain level of knowledge. The point is to enable you to use it in the real world.

So here I am, studying for my TSHOOT exam, which will complete my CCNP, and in the week before the exam comes 2 real world examples of how studying for the TSHOOT exam has helped me investigate and resolve the issue in hand. The next 2 posts will outline the problem, methodology used and how the TSHOOT exam materials directly related to resolving the problems.

Trouble Ticket #1 - The customer contacted us to state that SiteA no longer had access to the internet via it's default route. 

 Following the TSHOOT troubleshooting methodology I gathered as much relevant information as I could from the client. I then reviewed the information, eliminated the possible causes until I was down to a small   number of possible issues, I then formed my hypothesis, tested it and reviewed the outcome.

For this particular problem the customer stated that following the closure of one office and the migration to a new site access to the internet via it's default route was no longer possible. The customer supplied me their default route - 10.1.1.1 and that it originally routed via the now closed office. I also had the new network diagram outlining the new office and the appropriate routes. The diagram was essentially as follows:

SiteA Firewall ---> R1at SiteA ---> 'The core network' at R2 ---> R3 at SiteB - (trunk link)--> SiteB L3 switch ---> SiteB's ASA firewall --> INTERNET

First job was to see where the routing began to fail. So on to R1 and check the interfaces were up (yes) and then I moved on to the routing table to check for a route for 10.1.1.1. This was a static route pointing to our core network (for this article we'll call all of it R2). I performed a traceoute on R1 to 10.1.1.1 and it failed at R2.

On R2 I did the same, checked that interfaces were up and then checked the routing, This is where the first issue occurred. The static route on R2 was still pointing to the closed office. So the change was made. Static route was now pointing to R3 at SiteB.

I performed a traceroute to the default route from R1 again and it still didn't work. This time the trace stopped at R3.

On to R3 the routing table revealed that there was no route for 10.1.1.1 at all. After referring to the customer they advised me that the traffic had to go via a specific VLAN on the L3 switch so on R3 we needed to create VLAN10 and allowed VLAN10 on the trunk link to the L3 switch at SiteB. Finally the route for 10.1.1.1 was set to go via the SVI for VLAN10.

With this created I tested the route to 10.1.1.1 from R3. It still didn't work? With the information at hand I was able to formulate my hypothesis.

Was VLAN10 a) created on the L3 switch? and b) allowed on the trunk link?

Next I checked the state of the VLANs. VLAN10 was in place on R3, and was allowed on the trunk link to the L3 switch. The trunk link was up and protocol was up. Everything that I had admin access to was now properly configured and had been tested however the traceroute was still not returning. I put this to the customer who inspected their L3 Switch. They reported that VLAN10 had not been created after all on their equipment.

Once the customer applied their configuration on their equipment the route from SiteA to the ASA firewall on the LAN of SiteB worked. It was only after the issue was investigated that it became apparent the customer had neglected to account for the routing required for SiteA when they planned the closure of their old office. As a result none of the required configuration had been prepared before hand and SiteA had been cut off once the old office closed.

Trouble Ticket #1 - Closed.

As you can see on this occasion I was able to use a 'Divide and Conquer' trouble shooting approach. Starting L3 and then checking L1 and L2 when the traceroute failed. On R3 I was able to narrow the issue to L2 and formulate my hypothesis which was then tested.

CCNP - TSHOOT - Troubleshooting the Routers...

Continuing my TSHOOT methodology, in this post I'm going to look at troubleshooting the network between R4, R3, R2,R1 and out to the web server at 209.65.200.241.

Having not sat the TSHOOT exam the impression I'm getting is the Trouble Ticket (TT) will state something like Client1 cannot access [insert resource]. As such you need to identify the system the issue is on and the resolution.

Again I'll be referring to the actual  TSHOOT topology - here

Let's say Client 1 cannot access the web server at  209.65.200.241. Based on my previous article, also assume Client 1 can ping 10.1.4.5 at R4. In this post I will outline how I see myself troubleshooting the connectivity issue to the web server.

I'll want to prove connectivity to R1. So in turn I'll ping each interface on the way to R1 at 10.1.1.1 If at any point the ping is not returned then log in to the router that you were trying to ping and begin troubleshooting.

1) Are the interface on the link UP/UP? check that one of them is not admin'd down:
#sh ip interfaces brief

2) With interfaces up, is the L2 Frame-Relay topology correct? the diagram shows that the routers are connected via point-to-point links therefore I'm expecting to see a config similar to:
R4(config)#int s0/0/0
R4(config-if)#encapsulation frame-relay point-to-point
R4(config-if)#no shutdown
!
R4(config)#int s0/0/0.34 point-to-point
R4(config-if)#ip address 10.1.1.10 255.255.255.252
R4(config-if)#frame-relay interface dlci 403

3) Assuming L2 is fine move on the routing protocols. Things to check here are:
i - is route redistribution functioning correctly on R4? check the routing table, check config
ii - Are the adjacencies up? Check the neighbor tables/topology tables.
iii -Are the correct networks being advertised to the neighbor? check the ospf network statements
iv -  is there any authentication in place? if so it must match at each end
v - is the routing table including the correct route as per the network diagram? If the route is not in the routing table it won't be advertised out to the neighbor.
vi - Is the routing config command 'passive-interface default' set? if it is then updates will be suppressed by default. You need to either remove the command using #no passive-interface default
Or you specifically list the interfaces you want to participate in dynamic routing using:
#passive-interface default
#no passive-interface f0/0
#no passive-interface f0/1
and so on...

4) Up to R1 I think that would cover off all the major connectivity issues you'd be likely to get. If you still have problems look closer at R1.

On R1 check you can get to the next-hop outside of the network. If you can the config on R1 should be sound. If you can't it could indicate any of the following.
i - Check over the interfaces as per previous points. Ensure they are UP/UP. if they aren't then troubleshoot accordingly.
ii - Check routing/BGP - is the BGP peering up? if not check the config, if the update source being used is a Loopback address then you need the update-source command set in the BGP config. Check the IP's of the Peers, check the remote-as command is correct
iii - Is there a default route? I'd expect there to be one. Simple question. Is it pointing to the correct outbound next-hop? not in towards the LAN?
iv - Are there any access-lists? if so, what are they permitting and what are they blocking? Could the implicit deny  be causing an issue here?
v- Is the default route being correctly redistributed? I'm expecting to see something like:
R1(config)#router ospf 1
R1(config)#default-information originate
vi - Is NAT in operation? check which interface is inside, outside, check the ip nat source statements. Check any NAT pools that my be configured? if a pool is not configured then I'm looking for the 'overload' option in the command :
#ip nat source list 10 interface s0/0/1 overload
With out Overload then port address translation is not working and you'd only be able to have 1 host using NAT at any one time.


Based on the exam topology I'd hope that by running through my structured check list from L1, L2 to L3 to L4 and above that the issue in hand would be revealed.

When I'm in the exam I'll be taking my cue from the information in the TT. As such I might not need to bother with L1/L2 checks if the remote peer is responding as expected.

My next article is going to look at the IPv6 topology and look at how I might be troubleshooting problems there.

CCNP - ENTERPRISE - Considering the Client...

In the TSHOOT exam the bulk of the trouble tickets will pretty much be Client1 cannot access [insert resource here], identify the problem device and present a solution.

The TSHOOT topology is freely available from the Cisco website - here - and so we can take a look and start to suggest possible problems that can be presented. On it we have BGP, OSPF, EIGRP and redistribution, NAT, DHCP, IPv6 Tunnels, OSPFv3 and RIPng redistribution, we've got first hop redundancy and etherchannel, plus OSPF over Frame-Relay. Some pretty tasty stuff.

So, where to start? Well, lets say that Client1 can't access the web server at 209.65.200.241. As I'm looking at client issues assume that the config between the web server all the way in to the network to R4 is fine.

When I'm in the exam I'm going to be looking to ensure connectivity to Client1's default gateway and the DHCP server at R4 is sound. When I find that Client1 is unable to ping the default gateway or obtain a DHCP allocation here is what I'll be looking for:
1) Client1 IP Assignment - The L2/L3 topology diagram states that the address is obtained via DHCP (R4 at 10.1.4.5). On Client1 do >ipconfig - if the address is 10.2.1.4 move to ASW1, if the address is 169.X.X.X then DHCP is not working properly for some reason. In both cases move to ASW1

2) Interface VLAN membership - Client1 should be connected to int f1/0/1 - is it? if not then correct as appropriate. Interface f1/0/1 should be a member of VLAN 10 - does the VLAN exist? :
#sh vlan

Is int f1/0/1 a member of VLAN 10?:
#sh run int f1/0/1 OR #sh vlan

Does ASW1 have any port security?
#sh run int f1/0/1
if so what is configured here? Does any static MAC address match that of Client1? does the port limit number of MAC addresses per interface? Problem is here is if port security is incorrect. Otherwise move on

3) Is VLAN 10 allowed on the trunks?
Firstly prove int Po13 is an active Etherchannel and that interfaces f1/0/19-20 are members
#sh etherchannel summary - Trouble shoot accordingly if this is not the case.

Next is int Po13 a trunk link?
#sh interfaces trunk - if not then problem is here.

Next which VLAN's are allowed on the trunk links?
#sh interfaces po13 trunk - if VLAN10 is not here then you won't get to DSW1

Next, can you see any VLAN filtering in place? if so check that any permited ACL has the correct ip range otherwise traffic from 10.2.1.4 will not be allowed to use VLAN10. Also check the vlan access-map config ensure that the traffic allowed in the ACL is forwarded not dropped.

Do step 3 on both ASW1 and DSW1. If one side is not configured correctly then connectivity will fail.
If everything looks good you should be able to at least ping VLAN10's dfg at 10.2.1.1 on DSW1, move on.

4) Check routing on DSW1:
- DSW1 is an L3 switch that needs to send your DHCPDISCOVER to the DHCP server on a different network. As such ensure ip routing is enabled:
DSW1#sh run | begin ip routing
- Check L3 connectivity between DSW1 and R4 - Ping 10.1.4.5, check interfaces are UP/UP
DSW1#sh ip interface brief
- Check DSW1 int f1/0/1 is not admin'd down:

- If all looks good here check that the ip-helper command is applied to interface VLAN 10:
DSW1(config)#int vlan10
DSW1(config-if)#ip helper-address 10.1.4.5
(this will tell DSW1 to forward UDP broadcasts received on this interface)

5) You should now be at R4. If all of the above looks good then you should look above L2/L3 to DHCP or other factors.

6) DHCP configuration:
- Is DHCP running on R4? look for service dhcp in #sh run
- Are there any leases active? check - #sh ip dhcp bindings
Check the Hardware Address with that of Client1 to ensure the correct Client has an assignment. If you've discounted steps 1-4 I'd be surprised if this was the case.
- Check the DHCP address pool is correctly configured:
DSW1#sh run | begin ip dhcp
look for any kind of misconfig here be it incorrect subnet defined for the pool or an exclude address incorrect

7) At this point I'd expect to have uncovered an issue. If that's not case there's still some tactics I can try.
i) Kevin Wallace suggested Abort! - if you think you've narrowed the issue to a point but are unsure of the final solution, hit Abort, dive in to a different ticket and cross check the config. If the config is different the here is your fault. If it's the same then you can discount that area as the source of the problem and move on
ii) Let the question guide you - you've narrowed the issue to, lets say, DSW1 but are still unsure. Review the options available to you in part 2 and part 3 of the trouble ticket and use the information there to gleen if you are looking in the right place.

I think I've covered off the major issues that could be presented. I guess HSRP on DSW1 could be an issue but if DSW2 is the active router (#sh standby) then you should still be able to reach the DHCP server at 10.1.4.5?

Please let me know if I've missed a point or am miss guided in any way. Thanks

CCNP - ENTERPRISE - OSPF - Adjacency requirements

For an OSPF adjacency to form the 2 neighbors must agree on several parameters within the Hello Packet before the adjacency can form. These are:

i) Each must have a unique Route-ID
ii) Each must be in the same Area
iii) Authentication setting must match
iv) Timers must match.

One other parameter that must be agreed upon is the router priority for DR/DBR elections.

CCNP - ENTERPRISE - Manipulating Routing Updates - Route-map permissions

When compiling a Route-map(RM), you set an access control list (ACL) then use the Route-map to match addresses set out in that ACL to apply your chosen criteria in the Set field of the Route-map.

Now the question is given the combination of permit or deny statements in the ACL and the permit or deny statement of the Route-map what is the out come for a packet.

The following is what happens to a given packet when the permit or deny statements are considered:

ACL = Permit
RM= Permit
Result = Packet Permitted to proceed via the route-map. That's to say the packet is permitted to be permitted.

ACL = Deny
RM = Permit
Result = Packet Denied. The packet is denied from being permitted.

ACL = Permit
RM = Deny
Result = Packet Denied. The packet is permitted to be denied.

ACL = Deny
RM = Deny
Result = Packet PERMITTED. The packet is denied from being denied. If it isn't allowed to be denied, it must, therefore, be permitted.

Bit of a weird one to get your head round but it's an obvious trick to chuck in there when you're under pressure so keep an eye out.

CCNP - ENTERPRISE - IPv6 - IP address types

The following is a list of IPv6 address types. The high order bits are displayed and their function:

i) 001 = Global - 200
ii) 1111 1111 = Multicast - FF
iii) 1111 1110 11 = Site Local - FEC0
iv) 1111 1110 10 = Link Local - FE80
v) ::X:X:X:X = IPv4 compatible address, where the first 96 bits are set to 0 (hence the ::) and the remaining 32 bits are converted to hex from the IPv4 address

Other addresses include:
::1 or 0:0:0:0:0:0:0:1 = Loopback
::/128 = unspecified address which is essentially the DFG for IPv6, all the bits are set to 0.
IPv6 private addresses start - 1111 1110 1 - therefore both site local and link local are private

CCNP - ENTERPRISE - EIGRP - Adjacency Requirements

The following criteria need to be meet before an EIGRP adjacency will form:
i) Authentication (if in place)
ii) AS Number
iii) Source IP MUST be the primary address for the interface - secondary IP's will not result in the adjacency forming
iv) K values must match

N.B. - Timers do not have to match but they must be equal.
- Adjacency will flap if timers are mismatched.
- Therefore ensure you have a reliable time source.

CCNP - TSHOOT - Maintenance Models

The main models used in assisting with the maintenance of a network are as follows:

1. FCAPS - which is - Fault mgmt, Configuration mgmt, Accounting mgmt, Performance mgmt and Security mgmt. Defined by ISO (International Organisation for Standardisation).
2. ITIL - the IT Infrastructure Library, which is a collection of best practices. When implementing this PLEASE PLEASE PLEASE remember it is Best Practice not THE Practice. I've worked at companies who have idiots Service Delivery Mangers who think that to be 'compliant' they need to implement all of the guidelines regardless of how big the firm is or how suitable the process is to their enterprise </rant>
3. TMN - Telecommunications Management Network, the ITU-T's variant to FCAPS tailored to the telecommunications field.
4. Cisco Lifecycle Services - You should be familiar with this from your ROUTE and SWITCH studies Chpt1. Covering the PPDIOO - Prepare, Plan, Design, Implement, Operate, and Optimize life cycle.