Skip to content

By Administrator in All, Cisco, Routing protocols

Layer 3 separation with OSPF instances – conceptual overview

The main idea is that each provider router will be aware of all customer routes while each customer router will only learn routes relevant to their network. Separate OSPF instances are running for each customer, and are enabled on provider facing interface. One OSPF instance number 1 collects routes from all customers through redistribution. We will call it the super instance. Then, the routes are spread throughout the OSPF domain. The super instance is enabled only on provider-to provider links.

All links of the OSPF super instance are in area 0. To distinguish different customer routes when they are imported into the super instance, the routes are tagged. In this scenario, I’m tagging the routes with the OSPF instance ID they are imported from. To keep customer routes from leaking, each OSPF instance (except the super instance) imports only routes with a tag equal to the instance ID. If differently tagged routes are imported it will cause route leaking which may be desired or not. As a consequence, the IDs of OSPF instances running for the same customer, but on different sites, must be equal in order to receive OSPF updates.

This model solves the problem with route separation, but the problem with overlapping customer or provider networks remains. Since provider routers use only the main routing table to store routes, customer and provider subnets need to be unique. To circumvent this restriction we will be using the traditional NAT technique. The provider may allocate a separate subnet for that purpose, with each customer network mapped to one IP. It is configured as a Loopback IP address at the customer edge (CE) router, where NAT is performed. While customers still receive dynamic updates from other sites, for outgoing traffic passing through the provider network the source IP is swapped with the Outside Global address (loopback0). This process closely resembles MPLS behavior. Control plane information, e.g. routing is using non-NAT-ed addresses and data plane traffic is transported using NAT-ed addresses.

Summary of configuration steps

  • Create and configure OSPF super instance number 1 on all provider routers. It collects and carries routes from all customer sites. It is enabled on links connecting provider routers. All links are in area 0.
  • Create and configure per-customer OSPF instances on every provider edge (PE) router. Sites belonging to the same customer are running OSPF with the same instance IDs. The instances are enabled on the links facing the CE routers. The links may be in any area, even area 0, but for the sake of clarity I use the same OSPF areas on different sites.
  • Import routes from all other OSPF instances into the OSPF super instance. Tag the imported routes with a tag equal to the instance ID they are imported from.
  • Import routes from the OSPF super instance into other OSPF instances only when the route tag is equal to the instance ID.
  • Map on-site customer networks to a unique Loopback interface IP address at the CE router through NAT. This allows overlapping IP addressing schemes in provider and customer networks. The Loopback interface should be enabled for OSPF routing, so that it will be advertized throughout the provider and customer networks.

Note: No default route should be advertized by the PE routers. Otherwise all customer networks will have connectivity between each other due to classless IP routing. For the same reason customer OSPF areas cannot be stub or NSSA areas. For Internet connectivity another link should be used and default route should not be redistributed in OSPF.

Scenario network topology diagram

Network topology diagram

Network topology diagram

We assume that all routers are configured according to the above topology diagram, IP connectivity is established and OSPF instances are running. OSPF instance 1 is the super instance, OSPF instance 2 is servicing customer A and OSPF instance 3 is servicing customer B. We start the example by configuring route-maps and redistribution.

Router configuration and verification

PE1
!Configure route-map  to tag when redistributing into OSPF
!super instance from instance 2
!
PE1(config)#route-map TagCEA1 permit 10
PE1(config-route-map)#set tag 2
PE1(config-route-map)#exit
!
!Configure route-map to filter when redistributing into OSPF
!instance 2 from super instance
!
PE1(config)#route-map MatchCEA1 permit 10
PE1(config-route-map)#match tag 2
PE1(config-route-map)#exit
!configure redistribution
PE1(config)#router ospf 1
PE1(config-router)#redistribute ospf 2 route-map TagCEA1 subnets
PE1(config-router)#exit
PE1(config)#router ospf 2
PE1(config-router)#redistribute ospf 1 route-map MatchCEA1 subnets
!
!Configure route-map  to tag when redistributing into OSPF super
!instance from instance 3
!
PE1(config)#route-map TagCEB1 permit 10
PE1(config-route-map)#set tag 3
PE1(config-route-map)#exit
!
!Configure route-map to filter when redistributing into OSPF
!instance 3 from super instance
!
PE1(config)#route-map MatchCEB1 permit 10
PE1(config-route-map)#match tag 3
PE1(config-route-map)#exit
!configure redistribution
PE1(config)#router ospf 1
PE1(config-router)#redistribute ospf 3 route-map TagCEB1 subnets
PE1(config-router)#exit
PE1(config)#router ospf 2
PE1(config-router)#redistribute ospf 1 route-map MatchCEB1 subnets

Continue with PE2 configuration which is similar:

PE2
PE2(config)#route-map TagCEA2 permit 10
PE2(config-route-map)#set tag 2
PE2(config-route-map)#exit
PE2(config)#route-map MatchCEA2 permit 10
PE2(config-route-map)#match tag 2
PE2(config-route-map)#exit
PE2(config)#router ospf 1
PE2(config-router)#redistribute ospf 2 route-map TagCEA2 subnets
PE2(config)#router ospf 2
PE2(config-router)#redistribute ospf 1 route-map MatchCEA2 subnets
PE2(config)#route-map TagCEB2 permit 10
PE2(config-route-map)#set tag 3
PE2(config-route-map)#exit
PE2(config)#route-map MatchCEB2 permit 10
PE2(config-route-map)#match tag 3
PE2(config-route-map)#exit
PE2(config)#router ospf 1
PE2(config-router)#redistribute ospf 3 route-map TagCEB2 subnets
PE2(config)#router ospf 3
PE2(config-router)#redistribute ospf 1 route-map MatchCEB2 subnets

Verify routing table and connectivity on CEA2:

CEA2
CEA2#sh ip route


     10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O E2    10.0.1.0/24 [110/1] via 10.0.4.1, 00:00:15, FastEthernet0/0
O E2    10.0.0.1/32 [110/2] via 10.0.4.1, 00:00:15, FastEthernet0/0
C       10.0.4.0/24 is directly connected, FastEthernet0/0
C       10.0.0.5/32 is directly connected, Loopback0
O E2 192.168.0.0/24 [110/2] via 10.0.4.1, 00:00:15, FastEthernet0/0
C    192.168.1.0/24 is directly connected, FastEthernet0/1

CEA2#ping 192.168.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip
min/avg/max = 720/916/1448 ms
CEA2#

All routes are present and connectivity is established.

An unexpected problem

However, when we check CEB2’s routing table we stumble on an issue we didn’t anticipate:

CEB2
CEB2#sh ip route
O E2 192.168.10.0/24 [110/2] via 10.0.5.1, 00:00:58, FastEthernet0/0
     10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O E2    10.0.2.0/24 [110/1] via 10.0.5.1, 02:27:56, FastEthernet0/0
O E2    10.0.0.2/32 [110/2] via 10.0.5.1, 02:27:56, FastEthernet0/0
C       10.0.0.6/32 is directly connected, Loopback0
C       10.0.5.0/24 is directly connected, FastEthernet0/0
C    192.168.2.0/24 is directly connected, FastEthernet0/1
CEB2#

The route to the overlapping network 192.168.0.0/24 is missing. Let’s first check if CEB1 is advertizing the 192.168.0.0/24 subnet:

CEB1
CEB1#sh ip ospf database router self-originate

            OSPF Router with ID (10.0.0.2) (Process ID 1)

                Router Link States (Area 2)

  LS age: 507
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.2
  Advertising Router: 10.0.0.2
  LS Seq Number: 80000015
  Checksum: 0x153
  Length: 60
  Number of Links: 3

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 192.168.0.0
     (Link Data) Network Mask: 255.255.255.0
      Number of TOS metrics: 0
       TOS 0 Metrics: 1
......................
some parts skipped
......................
CEB1#

Indeed, the route is in the link-state database. Then, let’s check whether PE1 is receiving the update:

PE1
PE1#sh ip ospf database router 10.0.0.2

            OSPF Router with ID (10.0.2.1) (Process ID 3)

                Router Link States (Area 2)

  LS age: 708
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.2
  Advertising Router: 10.0.0.2
  LS Seq Number: 80000015
  Checksum: 0x153
  Length: 60
  Number of Links: 3

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 192.168.0.0
     (Link Data) Network Mask: 255.255.255.0
      Number of TOS metrics: 0
       TOS 0 Metrics: 1
......................
some parts skipped
......................

PE2 is receiving the update but it is not redistributed into OSPF instance 1:

PE1
PE1#sh ip ospf database | begin Process ID 1
......................
some parts skipped
......................

                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
10.0.0.1        10.0.0.3        1006        0x80000005 0x00563E 2
10.0.0.2        10.0.0.3        845         0x80000001 0x006630 3
10.0.0.5        10.0.0.4        1893        0x80000009 0x00206B 2
10.0.0.6        10.0.0.4        395         0x80000009 0x002861 3
10.0.1.0        10.0.0.3        489         0x8000000C 0x003D51 2
10.0.2.0        10.0.0.3        1006        0x80000009 0x004A45 3
10.0.4.0        10.0.0.4        141         0x8000000A 0x001A72 2
10.0.5.0        10.0.0.4        395         0x80000009 0x002368 3
192.168.0.0     10.0.0.3        1014        0x80000005 0x00AC74 2
192.168.1.0     10.0.0.4        1900        0x80000009 0x001818 2
192.168.2.0     10.0.0.4        402         0x80000009 0x001F0F 3
PE1#

There is one tagged entry for 192.168.0.0/24 (with a tag of 2) from the other customer routing instance. It is the OSPF multi-instance behavior to blame for the missing route. More details will be given at the end of the scenario. For now it is enough to remember that the route, which is received first is installed in the routing table. And in case you wonder if load balancing is possible, the answer is “no”.

Solution and consequences

Obviously, some information is lost. The route appears only once in the main routing table and once in the OSPF super instance link-state database. So the other customer (B) , which imports only routes with a tag equal to 3 will not receive it.

To resolve the issue, we use a VRF technique. The overlapping subnet is tagged with a different tag and then it is imported in both customer routing instances.

The necessary changes on the PE1 router:

PE1
ip access-list standard Net192.168.0.0
 permit 192.168.0.0 0.0.0.255

route-map TagCEA1 permit 5
 match ip address Net192.168.0.0
 set tag 23
!
route-map TagCEA1 permit 10
 set tag 2
!

The necessary changes on the PE2 router:

PE2
route-map MatchTagCEA2 permit 10
match tag 2 23
route-map MatchTagCEAB2 permit 10
match tag 3 23

In order to verify the changes:

CEB2
CEB2#sh ip ospf database | begin External
                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
10.0.0.2        10.0.4.1        1848        0x80000004 0x005041 3
10.0.2.0        10.0.4.1        1342        0x8000000C 0x003456 3
192.168.0.0     10.0.4.1        140         0x80000008 0x009685 23
CEB2#
CEA2
CEA2#sh ip ospf database | begin External
                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
10.0.0.1        10.0.5.1        661         0x80000002 0x00454F 2
10.0.1.0        10.0.5.1        661         0x80000002 0x003A5B 2
192.168.0.0     10.0.5.1        43          0x80000001 0x009D84 23
CEA2#

Now, the route to 192.168.0.0/24 appears at both sites with a tag of 23.

Note: Interesting behavior is observed when 192.168.0.0/24 is down. If it is the 192.168.0.0/24 network on CEA1 that is down, then the 192.168.0.0/24 route from CEB1 is immediately redistributed into the OSPF super instance and gets a tag 3. CEA2‘s instance is not importing this route and connectivity to the subnet is lost, as it should be. CEB2‘s OSPF instance is importing routes tagged with 3 and the route stays. But when 192.168.0.0/24 on CEA1 is up again it will not be redistributed into OSPF 1 and connectivity to network from CEA2 will not be restored until CEB1 fails. If we tag 192.168.0.0/24 from both customers with tag 23, then the route will be advertized until there is at least one customer network up. This is the punishment for routing information being lost.

To implement the second scenario, I tag the route for 192.168.0.0/24 (tag 23) from CEB1 as well:

On PE1:

PE1
PE1(config)#route-map TagCEB1 permit 5
PE1(config-route-map)#match ip address Net192.168.0.0
PE1(config-route-map)#set tag 23

No changes are necessary on PE2. At the end of the article we’ll extend the scenario to account for the missing information. Now, we leave the control plane and continue with data plane. As seen, we’ve assured routing information delivery, but data traffic is still using overlapping subnets. To solve the data plane problem, NAT is employed. If now we try a ping from CEB1 to CEB2’s IP address 192.168.2.1 with source 192.168.0.10 the result is:

CEB1
CEB1#ping 192.168.2.1 source 192.168.0.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.10
.....
Success rate is 0 percent (0/5)
CEB1#

Let’s make the following changes to CEB1’s configuration:

CEB1
CEB1(config)#ip access-list standard NAT
CEB1(config-ext-nacl)#permit ip 192.168.0.0 0.0.255.255
CEB1(config-ext-nacl)#exit
CEB1(config)#int fa0/1
CEB1(config-if)#ip nat inside
CEB1(config-if)#int fa0/0
CEB1(config-if)#ip nat outside
CEB1(config-if)#exit
CEB1(config)#ip nat inside source list NAT interface
loopback 0 overload

If we try try the above ping command again, the result will be a 100% success.

To verify that NAT is employed:

CEB1
CEB1#sh ip nat translations
Pro Inside global    Inside local     Outside local    Outside global
icmp 10.0.0.2:9      192.168.0.10:9   192.168.2.1:9    192.168.2.1:9
CEB1#

Connectivity is established but for good design practice we need to setup NAT on all customer routers (not shown).

Back to the missing route problem

Below is presented one possible solution to the problem. May be it is not the best, but it is a way out and will complement the model presented. Since routing information for the overlapping subnet is not always correct, we choose a unique piece of information for each customer to track. Of course, this is the Loopback0 interface address, which participates in customer OSPF routing. We still cannot track the state of each route in the customer network, but at least we can be sure that the CE router is up and OSPF is running. The mechanism acts as follows. On CEA2 we track reachability to CEA1‘s Loopback0 interface by a track object. If the Loopback0 interface is reachable, CEA2 will use the OSPF external route to 192.168.0.0/24. If the track object is DOWN, a static route to Null is installed for the 192.168.0.0/24 network. It will be preferred over the OSPF route and all packets will be locally dropped and not transit the provider network. The inverse behavior of the static route installation is achieved by using a second track object. It is of list type and negates the state of the first object.

Summary of the steps to perform on CEA2:

  • Configure track object No 1 to track reachability to CEA1 Loopback0 interface (10.0.0.1).
  • Configure track object No 2 which is a Boolean list containing only one item. Add object No 1 to the list and negate its state. Because the list object has only one item the state of the object is determined by the state of that item which is the opposite of object No 1.
  • Configure static route to Null for 192.168.0.0/24 depending on track object No 2.

Implementation and verification steps:

CEA2
!first track object
!
CEA2(config)#track 1 ip route 10.0.0.1/32 reachability
CEA2(config-track)#delay down 1 up 1
CEA2(config-track)#exit
!second track object
CEA2(config)#track 2 list boolean and
CEA2(config-track)#object 1 not
CEA2(config-track)#delay up 1 down 1
!speed up convergence
CEA2(config)#track timer ip route 1
!static route for 192.168.0.0/24
CEA2(config)#ip route 192.168.0.0 255.255.255.0 null0 track 2
CEA2(config)#exit
CEA2#

Verify state of track object No 1:

CEA2
CEA2#sh track 1
Track 1
  IP route 10.0.0.1 255.255.255.255 reachability
  Reachability is Up (OSPF)
    1 change, last change 00:03:36
  Delay up 1 sec, down 1 sec
  First-hop interface is FastEthernet1/0
  Tracked by:
    Track-list 2
CEA2#

The route to 10.0.0.1 is present, state is up.

Verify the state of track object 2:

CEA2
CEA2#sh track 2
Track 2
  List boolean or
  Boolean AND is Down
    3 changes, last change 00:12:27
    object 1 not Up
  Delay up 1 sec, down 1 sec
  Tracked by:
    STATIC-IP-ROUTING 0
CEA2#

State is down.

Verify routing table:

CEA2
CEA2#sh ip route

     10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
O E2    10.0.1.0/24 [110/1] via 10.0.4.1, 02:34:35, FastEthernet1/0
O E2    10.0.0.1/32 [110/2] via 10.0.4.1, 00:13:43, FastEthernet1/0
C       10.0.4.0/24 is directly connected, FastEthernet1/0
C       10.0.0.5/32 is directly connected, Loopback0
O E2 192.168.0.0/24 [110/2] via 10.0.4.1, 00:13:40, FastEthernet1/0
C    192.168.1.0/24 is directly connected, FastEthernet1/1
CEA2#

The route to 192.168.0.0/24 is obtained through OSPF.

Now we shut down CEA1 Loopback0 interface and issue the same commands:

CEA2
CEA2#sh track 1
Track 1
  IP route 10.0.0.1 255.255.255.255 reachability
  Reachability is Down (no route)
    4 changes, last change 00:00:08
  Delay up 1 sec, down 1 sec
  First-hop interface is unknown
  Tracked by:
    Track-list 2
CEA2#

The state is DOWN.

CEA2
CEA2#sh track 2
Track 2
  List boolean or
  Boolean OR is Up
    4 changes, last change 00:00:54
    object 1 not Down
  Delay up 1 sec, down 1 sec
  Tracked by:
    STATIC-IP-ROUTING 0
CEA2#

The state of the second object is UP and a static route to null for 192.168.0.0/24 is installed:

CEA2
CEA2#sh ip route
Gateway of last resort is not set

     10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O E2    10.0.1.0/24 [110/1] via 10.0.4.1, 02:39:51, FastEthernet1/0
C       10.0.4.0/24 is directly connected, FastEthernet1/0
C       10.0.0.5/32 is directly connected, Loopback0
S    192.168.0.0/24 is directly connected, Null0
C    192.168.1.0/24 is directly connected, FastEthernet1/1
CEA2#
CEA2#sh ip ospf database | begin Type-5
                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
10.0.0.1        10.0.5.1        156         0x80000001 0x00474E 2
10.0.1.0        10.0.5.1        156         0x80000001 0x003C5A 2
192.168.0.0     10.0.5.1        166         0x80000001 0x009D84 23
CEA2#

OSPF route selection in multi-instance OSPF

OSPF route selection dictates that intra-area routes are preferred over inter-area routes, which are preferred over external routes. However ,this rule applies to routes learned via the same OSPF instance (Ask Cisco). Routes from different instances are compared their administrative distances. By default OSPF instances will have the same administrative distances. And if a route is installed via one OSPF instance, it will NOT be overwritten by another OSPF instance unless the route is first deleted from the routing table by the instance that installed it. So the route selection becomes non-deterministic as observed in the above example. The first to come is installed, preventing all routes for the same prefix, which are received from other OSPF instances, from being installed. In order to achieve load balancing, multiple routes to the same prefix should be received via the same OSPF instance (which is not the case in this scenario). This behavior introduces a great potential for rooting loops. External routes form one OSPF instance may be installed in the routing table instead of internal routes, provided they are received earlier. This is not a concern in this scenario for customer addresses are only used in the customer control plane. Routing and data traffic in provider network is using NAT-ed IP addresses.

Tags: , , , , ,

Comment Feed

No Responses (yet)



Some HTML is OK

or, reply to this post via trackback.