{"id":1595,"date":"2009-03-22T13:14:46","date_gmt":"2009-03-22T12:14:46","guid":{"rendered":"http:\/\/www.corelan.be:8800\/index.php\/2009\/03\/22\/juniper-screenos-defeating-ibgp-full-mesh-requirement-using-route-reflectors-and-confederations\/"},"modified":"2009-03-22T13:14:46","modified_gmt":"2009-03-22T12:14:46","slug":"juniper-screenos-defeating-ibgp-full-mesh-requirement-using-route-reflectors-and-confederations","status":"publish","type":"post","link":"https:\/\/www.corelan.be\/index.php\/2009\/03\/22\/juniper-screenos-defeating-ibgp-full-mesh-requirement-using-route-reflectors-and-confederations\/","title":{"rendered":"Juniper ScreenOS : defeating iBGP full mesh requirement using route reflectors and confederations"},"content":{"rendered":"

As explained in one of my earlier posts<\/a>, one of the requirements to successfully setup and operate an iBGP configuration is that all iBGP clients need to have a BGP connection to all other iBGP clients. (= full mesh). This is required because an iBGP device only exchanges information about its own networks and it does not pass on BGP updates from other peers to other peers.<\/p>\n

Suppose you have 4 devices in the same AS, exchanging prefix information over BGP, then you would need 6 iBGP connections.<\/p>\n

\"image\"<\/a><\/p>\n

If you add one more router in the AS, you need 4 more iBGP connections <\/p>\n

\"image\"<\/a> <\/p>\n

In other words, adding more devices increases the number of peers exponentially. It is clear that this does not scale very well. Of course, you can use peer-groups to somewhat simplify the configuration of the peers, but you still need to configure neighbors to all other routers on all devices.<\/p>\n

Basically, the setup on each device (screenos) would look something like this :<\/p>\n

\n
set vrouter trust-vr\nset protocol bgp 65500\nset enable\nset neighbor peer-group "iBGPPeers"<\/span>\nset neighbor peer-group "iBGPPeers"<\/span> remote-as<\/span> 65500\nset neighbor peer-group "iBGPPeers"<\/span> md5-authentication "iBGPTest"<\/span>\nset neighbor 192.168.0.30 peer-group iBGPPeers\nset neighbor 192.168.3.10 peer-group iBGPPeers\nset neighbor 192.168.21.15 peer-group iBGPPeers<\/pre>\n<\/div>\n
so you basically need to create a peer group, set the parameters that will be applied to all peers in this group (remote-as and md5-authentication parameters) and then you define the peers into the peer group.  If you are using route-maps, you can either define the route-map in the peer group, and\/or directly on a specific peer.  If you define a route-map directly on a peer, it will take precedence over the global route-map in the peer-group.<\/p>\n
Nice, but using peer-groups only limits the number of lines of screenos statements, it does not defeat the full mesh requirement.  Some critics might even say that using peer-groups makes it more difficult when you are using multiple route-maps.<\/p>\n
A lots of work, risk of making mistakes\u2026. there are numerous reasons to consider when designing your network.  But if using iBGP is your only option, how can you make your life easier ?<\/p>\n
There are 2 solutions. You can use route reflectors and confederations to decrease the number of peer relationships you need to configure and maintain. Both solutions have some drawbacks too, but we\u2019ll talk about that when we get there.<\/p>\n
 <\/p>\n
Route reflectors<\/h3>\n
A route reflector is a router that acts as a hub between various iBGP clients. These clients peer with the RR (Route reflector) and exchange information with it.  The RR reflects the information to other BGP peers (iBGP and eBGP)<\/p>\n
Route reflectors introduce the concept of \u2018clients\u2019 and \u2018non-clients\u2019.  When a host is connected to a RR, it becomes a client.  The RR and the clients now form a cluster.  Peers of the RR that are not part of the cluster are non-clients.<\/p>\n
Non-clients still need to be configured to form a full mesh with the RR and the other iBGP speakers (but not with the other clients that are part of the RR cluster)<\/p>\n
So in other words, if we look back at the example of 5 routers, and now assume that router5 is a RR, and router 2 and 4 are RR clients, then the iBGP setup would look like this :<\/p>\n
\"image\"<\/a> <\/p>\n
router 1 and 3 now only have to peer with each other and with the RR.  RR clients 2 and 4 only need to peer with router5 (the RR).   So instead of having to configure 10 iBGP peers, you only need to set up 5 relationships.<\/p>\n
The drawbacks of using a RR are clear : the RR is now a sinle point of failure and the RR process will put somewhat more load on the router.  Of course, you can set up multiple RR\u2019s for added redundancy (you just need to set up all clients and non-clients to peer with all RRs)\u2026 and as long as you don\u2019t have thousands of routes exchange via BGP, the extra load should not be a big issue either.  And if you have more routes, you may need to reconsider the design and split up the network (AS)  into smaller networks (ASs) and use eBGP to exchange information between the various networks.<\/p>\n
Example :<\/p>\n
\"image\"<\/a><\/p>\n
(Note : a RR will only reflect the routes within the same AS. So if you peer RR\u2019s from multiple ASs with each other using eBGP, you will still need to redistribute prefixes from ebgp into ibgp on the RRs in order to get the information exchanged to the other routers in the AS.)<\/em><\/p>\n
Route reflectors will preserve all iBGP attributes, however some implementations still allow attributes to be changed or filtering to be applied.  I would not recommend changing attributes unless you really know what you are doing. If, for example, you start playing with the next_hop attribute, you may introduce a routing loop. On the other hand, it may be nice to be able to play with the local_pref or med values in order to prefer one path over another.<\/p>\n
It\u2019s also good to know that changing from a topology without a RR to a RR topology does not introduce a major design or configuration change. AS numbers still remain the same. You only need to set up the RR, set up the RR clients, and then you can start removing iBGP peer relationships.<\/p>\n
To a non-client, the RR just acts as a regular BGP peer, so even if the non-client is not RR aware of cannot handle RR configurations, as long as it can speak iBGP it will work. It will never know that the peer is a RR.<\/p>\n
This is how a RR network is set up in screenos :<\/p>\n
1. The route reflector<\/u><\/p>\n
\n
set vrouter trust-vr\nset proto bgp\nset reflector\nset reflector cluster-id 1<\/pre>\n<\/div>\n
That\u2019s it. Enable the reflector and set the cluster-id (to 1 in this example).  You can get the reflector settings using the \u201cget vrouter trust-vr proto bgp reflector\u201d statement :<\/p>\n
\n
get vrouter trust-vr proto bgp reflector\n\nRoute reflector:       enable\nCluster ID<\/pre>\n<\/div>\n
Now change the iBGP relationship with the RR clients (and add a statement to make them reflector client) :<\/p>\n
\n
set vrouter trust-vr\nset proto bgp\nset neighbor <ip_of_RR_client> reflector-client<\/pre>\n<\/div>\n
(if you were already using peer-groups, you can set the reflector-client configuration on the peer-group and it will be applied to all neighbors)<\/div>\n
 <\/div>\n
2. RR clients<\/u><\/p>\n
There\u2019s nothing special to configure on the RR clients. Just set up the iBGP peer relationship with the RR and that\u2019s it. You can remove the peer relationships with the other RR clients and with the non-RR clients.  RR clients only need to talk to the RR.<\/p>\n
When the client has been configured to peer with the RR, it becomes a reflector client. You can get query this info by running the following command on the client:<\/p>\n
\n
get vrouter trust-vr protocol bgp neighbor <ip of RR> | include reflector\n\n    reflector client: yes<\/pre>\n<\/div>\n
On the RR, you can also get the same info by running :<\/p>\n
\n
get vr trust-vr proto bgp neighbor 192.168.0.127\n\npeer: 192.168.0.127,  remote AS: 65000, admin status: enable, PeerID 2, CBAfi 0\ntype: IBGP\nconnection state: ACTIVE, connection id: 0 retry interval: node default<\/span>(120s), cur retry time 120s\nconfigured hold time: node default<\/span>(20s), configured keepalive: node default<\/span>(6s)\ndesignated local IP: n\/a\n--------------------------------------------------------------------------------------\nFor address family: IPv4 unicast\nroute map(ipv4) in<\/span> name: , route map(ipv4) out<\/span> name: \nweight(ipv4): 100 (default<\/span>)\nself as<\/span> next hop(ipv4): disable\nsend default<\/span> route to peer(ipv4): disable\nignore default<\/span> route from peer(ipv4): disable\nsend community path attribute(ipv4): no\nreflector client(ipv4): yes\nconfigured adv-interval(ipv4): default<\/span>(5s)\n--------------------------------------------------------------------------------------\nFor address family: IPv6 unicast\nroute map(ipv6) in<\/span> name: , route map out<\/span>(ipv6) name: \nweight(ipv6): 100 (default<\/span>)\nself as<\/span> next hop(ipv6): disable\nsend default<\/span> route to peer(ipv6): disable\nignore default<\/span> route from peer(ipv6): disable\nsend community path attribute(ipv6): no\nreflector client(ipv6): no\nconfigured adv-interval(ipv6): default<\/span>(5s)\n--------------------------------------------------------------------------------------\nNeighbor Capabilities: \n  Route refresh: advertised \n  Address family IPv4 Unicast:  advertised \nforce reconnect is<\/span> disable\ntotal messages to peer: 234, from peer: 206 \nupdate messages to peer: 30, from peer: 5\nTx queue length 0, Tx queue HWM: 5\nroute-refresh messages to peer: 0, from peer: 0 \ncurrent packets from peer number: 1, length: 21 \nlast reset 00:29:24 ago, due to BGP recv Notification(Cease: Admin stopped)(code 6 : subcode 3)\nnumber of total successful connections: 5\npeer down elapsed time: 1 minutes 53 seconds<\/pre>\n<\/div>\n
(the RR client is at 192.168.0.127, the RR is 192.168.0.8).  In fact, this client is a linux router running quagga.  <\/p>\n
As opposed to screenOS, on Cisco (or quagga) you do need to specify that the client is a route reflector. You can set up the reflector-client using the following commands :<\/p>\n
\n
router bgp 65000\n   bgp router-id 1.1.1.1\n   neighbor <ip_of_rr> remote-as<\/span> 65000\n   neighbor <ip_of_rr> route-reflector-client<\/pre>\n<\/div>\n
In quagga (and Cisco), you can verify that the client is a rr-client using the sh ip bgp neighbors command :<\/p>\n
\n
[root@linuxrouter-1 ~]# telnet localhost 2605\nTrying 127.0.0.1...\nConnected to localhost.\nEscape character is<\/span> '^]'<\/span>.\n\nHello, this<\/span> is<\/span> Quagga (version 0.99.10).\nCopyright 1996-2005 Kunihiro Ishiguro, et al.\n\n\nUser Access Verification\n\nPassword: \nlinuxrouter-1> en\nPassword: \n\n\nlinuxrouter-1# sh ip bgp neighbors\nBGP neighbor is<\/span> 192.168.0.8, remote AS 65000, local AS 65000, internal<\/span> link\n  BGP version 4, remote router ID 192.168.0.8\n  BGP state = Established, up for<\/span> 00:04:34\n  Last read 00:00:03, hold time is<\/span> 20, keepalive interval is<\/span> 6 seconds\n  Neighbor capabilities:\n    4 Byte AS: advertised\n    Route refresh: advertised and received(old & new<\/span>)\n    Address family IPv4 Unicast: advertised and received\n  Message statistics:\n    Inq depth is<\/span> 0\n    Outq depth is<\/span> 0\n                         Sent       Rcvd\n    Opens:                  1          1\n    Notifications:          0          0\n    Updates:                1          6\n    Keepalives:            47         45\n    Route Refresh:          0          0\n    Capability:             0          0\n    Total:                 49         52\n  Minimum time between advertisement runs is<\/span> 5 seconds\n\n For address family: IPv4 Unicast\n  Route-Reflector Client\n  4 accepted prefixes\n\n  Connections established 1; dropped 0\n  Last reset never\nLocal host: 192.168.0.127, Local port: 53411\nForeign host: 192.168.0.8, Foreign port: 179\nNexthop: 192.168.0.127\nNexthop global: fe80::a00:27ff:fe6f:9ac\nNexthop local: ::\nBGP connection: non shared network\nRead thread: on  Write thread: off<\/pre>\n<\/div>\n
 <\/div>\n
 <\/div>\n
3. non-RR clients<\/u><\/p>\n
There\u2019s nothing special to configure. Just remove the peer relationships with the RR clients. non-RR clients only need to peer with each other and with the RR.<\/p>\n
 <\/p>\n
 <\/p>\n
Confederations<\/h3>\n
BGP confederations offer a second way to limit the number of peer relationships in a iBGP environment. This technique is based on the concept of breaking an AS into smaller sub-ASs.  Peers in the sub-AS need to form a full mesh. Information between multiple sub-ASs is exchanged via eBGP (so you can dedicate one router in one sub-AS to exchange information with one router in the other sub-AS). To the outside world, the entire set of sub-ASs still look like one AS.<\/p>\n
Although eBGP is used between sub-ASs, BGP attributes are preserved while crossing the sub-AS boundary. So attributes such as med, local_pref, next_hop are not changed (as if you were doing iBGP)<\/p>\n
There are 2 major drawbacks in this scenario : <\/p>\n
1. Migrating from iBGP full mesh to confederations require a major reconfiguration of all routers<\/p>\n
2. BGP uses the shortest path to determine the best route. Because an AS can consist of multiple sub-ASs, but looks like one AS path hop to external BGP peers, you can get suboptimal routing. (after all, the sub-ASs do not influence the AS path length to external BGP peers). You can overcome this by manually setting the local_pref and med attributes on the eBGP peers.<\/p>\n
This is how confederations are set up in screenos :<\/p>\n
If BGP was already running, you need to break it down entirely. After all, you will need to put the bgp instance in a new AS (a sub-AS)<\/p>\n
When BGP is entirely gone from the config, you need to set 3 different AS numbers :<\/p>\n