Take a look at Figure 1. We'll explain more of the details below, but note the "Home Dialup User". He's connected to AOL, which is served by ANS (AOL actually owns ANS). We're using 10.10.20.0/24 as an example.

The 10.10.x.x IP addresses are often used in examples because they're "reserved" space. Most networks will "filter" the RFC 1918 reserved space (10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16), so people use them in examples because they don't get you into too much trouble if you accidentally try to use them (sort of like the film industry's yyy-555-xxxx phone number convention).

In this example, the reason that an AOL dialup user can send a packet to 10.10.20.1 (for example) is that the ISP (AS 64512) advertised that route to the two upstream providers (AS 4969 and AS 701), who in turn advertised that route to AS 690 (ANS, which provides IP service for AOL).

Every IP address that you can get to on the Internet is reachable because someone, somewhere, has advertised a route that covers it. The corollary to this is that if there is not a generally-advertised route to cover an IP address, no one on the Internet will be able to reach it.

HARDWARE AND SOFTWARE FOR SPEAKING BGP

I recommend using Cisco routers (for many reasons). In particular, the Cisco implementation of BGP is relatively easy to use, get examples for, and debug - and there's a huge community of routing engineers that's familiar with the Cisco implementation and algorithms (there's much that isn't specified in the RFCs and is left up to the vendor to decide). Cisco's online documentation (UniverCD) isn't the best (it lacks a large number of case studies) but is a very good learning tool.

PC-compatibles using gated are either the second- or third-largest community of BGP-speaking computers. You can build cheap PC routers that route Ethernet and t1 and have more than enough CPU and memory to handle all the routes you'd need for quite some time - but you've then got hardware that's not really as tested or reliable as a Cisco or Bay router. Trust me on this - the cost savings is usually not worth doing it this way. (Apologies to Riscom and ET, the leading vendors of T1 cards-for- PCs).

Bay routers are the second-largest community of BGP-speaking boxes - but we're talking about a very small percentage of the number of BGP- speaking Ciscos out there. Bay is cheaper than Cisco; pretty responsive to customers (though Cisco is as well); and almost all configuration is done through a GUI (windowing) interface that drives most routing engineers nuts. Bay claims they're working on a command-line interface, (BCC, or "Blatant Cisco Clone"), but in the mean time most are throwing money at Cisco. (It's much easier to debug BGP or other routing problems from a telnet session or over the phone than it is to have to guide someone through a GUI to examine or reconfigure a router). On the other hand, the Bays do have a better architecture and are finally showing themselves to be more or less as stable as Ciscos. What I've seen of BCC looks quite promising, and I promise to retract in print my slam of Bay when their command line interface looks featureful, fast, and solid.

We're going to talk about Cisco routers in these documents (and in this document in particular).

PEERING SESSIONS AND ASNs: PART I

BGP-speaking routers exchange routes with other BGP-speaking routers via peering sessions. At a technical level, this is what it means to "peer with someone". A snippet of a Cisco "BGP clause" is:

router bgp 64512
 neighbor 207.106.127.122 remote-as 701
 (omitted lines)
 neighbor 137.39.10.46 remote-as 4969
 (omitted lines)

In order to bring up a "peering session", all you need to do is have that one line. In this example, 137.39.10.46 is the remote IP address of a UUNET router (UUNET is ASN 701). Remote, that is, with respect to the customer's router. 207.106.127.122 is the remote IP address of a Net Access router (Net Access is ASN 4969). See Fig 1 for a diagram of the network layout used in this example.

In practice, however, you almost always use more than that one line to tell BGP how to exchange routes with that "neighbor" via that "peering session". A typical "neighbor clause" is:

router bgp 64512
 (omitted lines)
 neighbor 207.106.127.122 remote-as 4969
 neighbor 207.106.127.122 next-hop-self
 neighbor 207.106.127.122 send-communities
 neighbor 207.106.127.122 route-map prepend-once out
 neighbor 207.106.127.122 filter-list 2 in
 (omitted lines)

WHAT DO YOU DO WITH BGP?

PEERING SESSIONS

Every time a neighbor session comes up, each router will evaluate every BGP route it has by running it through any filters you specificity in the "neighbor" clause. Any routes that "pass" the filter are sent to the remote end.

While the session is up, "BGP Updates" will be sent from one router to the other each time one of the routers knows about a new BGP route or needs to "withdraw" a previous announcement ("promise").

The "sho ip bgp summ" command will show you a list of all peering sessions:

brain.netaxs.com#sho ip bgp summ
BGP table version is 1159873, main routing table version 1159873
44796 network entries (98292/144814 paths) using 9596344 bytes of memory
16308 BGP path attribute entries using 2075736 bytes of memory
12967 BGP route-map cache entries using 207472 bytes of memory
16200 BGP filter-list cache entries using 259200 bytes of memory

Neighbor      V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State
205.160.5.1   4  6313       0       0        0    0    0 never    Active
207.106.90.1  4 64514 1145670  237369  1159873    0    0 4d03h
207.106.91.5  4 64515    6078    5960  1159869    0    0 4d03h
207.106.92.16 4 64512    6128    6782  1159870    0    0 4d03h
207.106.92.17 4 64512    5962    6894  1159870    0    0 10:08:46
206.245.159.17  4  4231  161072  276660  1159870    0    0 2d05h
207.44.7.25   4  3564    6109  310292  1159867    0    0 22:40:50
207.106.33.3  4 64513  164708  724571  1159866    0    0 3d23h
207.106.33.4  4  3564    6086  274182  1159853    0    0 4d03h
207.106.127.6 4  6078    5793  310011  1159869    0    0 2d03h

Most of it is pretty self-explanatory; briefly:

More on all of this below.

eBGP vs. iBGP

The major difference between eBGP and iBGP is that eBGP tries like crazy to advertise every BGP route it knows to everyone - you have to put "filters" in place to stop it from doing so. iBGP is actually pretty difficult to get working because it tries like crazy not to redistribute routes - in fact, all iBGP-speakers inside your network have to peer with all other iBGP "speakers" in order to make it work. This is called a "routing mesh" and, as you can imagine, is quite a mess. If you have 20 routers, each router has to peer with every other router. The solution to this is "BGP confederations", also a topic for a future document.

Also, iBGP has major drawbacks as an IGP. The main one is the necessity to "peer up" every set of routers in your network (or in one POP if you're using confederations). Protocols like OSPF and IS-IS just "find" each other over serial and Ethernet interfaces (they're "broadcast" protocols). This can be a pain (you don't want to accidentally merge your IGP with a customer's or peer's) but turning off broadcasting on certain ports is easier than turning on peering sessions between a new router and every other router on your network. Also, iBGP doesn't do as good a job at "convergence" (closing the gap and re-routing around failed network segments) as OSPF and IS-IS.

BGP AND THE SINGLE-HOMED

And if you have one upstream provider, it's almost guaranteed that you are using sub-allocations (CIDR delegations, to be precise) of their larger IP blocks ("aggregates"). In this case your provider is not going to advertise your more "specific" routes because:

AS-PATHS

The AS-PATH is useful for a number of reasons:

AS-PATH LENGTH AND BGP ROUTE SELECTION

See Fig 2 for a sample list of routes from an actual BGP routing table - and further explanation. Notice, though, the >'s to the left of the some of the routes. The ">" indicates the route that the router currently thinks is "best" when there are multiple choices.

Fig 2.

A SNIPPET OF A BGP ROUTING TABLE

COMING SOON TO A TUTORIAL NEAR YOU.

AS-PATH ACCESS LISTS (FILTERS)

ip as-path access-list NNN permit regexp

ip as-path access-list NNN deny regexp

Fig 3

Regexp characters:

NNN   match the characters NNN (where each digit of NNN is
from 0-9)
^     match the beginning of a string
$     match the end of a string
_     match any of {space, beginning of a string, or end of a string}
_NNN_ match the "word" or "distinct number" NNN.  Thus, the regexp
      "_1_" will match the string "3561 1 64000" but not "3561".
      (The problem is that if you don't anchor NNN with "_"s on 
      either side, you might match something you don't really want to).

(regexp)  enclosing another regexp in parens
      means that the appearance of that regexp is optional
*     the * operator means that the previous regexp can be matched 
      0, 1, 2, or any number of times.  To be safe, only use * in 
      conjunction with parens.
      Thus, (regexp)* matches the regexp inside the parens 0 or
      any number of times.

[char1char2char3]   matches any one of char1, char2, char3, etc...
                    Each charN expression can be an actual
                    number or other
                    symbol, or a range (i.e. 0-9, a-z).

If you want to match any of the special symbols, you can escape them by
putting a \ in front of them.  The only special symbols you'll want to 
escape when matching against AS-PATHs are the parens, which pop up in 
AS-PATHs when you use BGP confederations.

Important note: On Ciscos, regexps are matched against the AS-PATH as if the whole thing is a string, not a sequence of numbers. Thus, as you'll see below, you need to enclose ASNs within underscores to be sure of matching only the ASN you're looking for.

How do access-lists work? When used as a filter, each route is passed through the access-list. Each rule is listed in the order it will be applied. Once a route has been matched by any rule, the decision on whether to pass the route through the filter or to drop it (and thus not let it pass) is made immediately, and no further rules are processed.

Example 1:

ip as-path access-list 1 permit .*
ip as-path access-list 1 deny .*

Example 2:

ip as-path access-list 2 deny .*

Example 3:

ip as-path access-list 3 permit ^$
ip as-path access-list 3 deny .*

If you have these three as-path access-lists installed and remember their numbers you'll save yourself a lot of time you'd otherwise spend searching online or through config files to find where you put your "send everything"; "send nothing"; or "send only my routes" filter.

Remember: BGP between different ASNs (eBGP) will, by default, cause a router to redistribute every BGP route that the router knows about. This could lead to VERY BAD THINGS happening. (If you redistributed all of Sprintlink's routes into UUNET, a portion of UUNET could start sending all of its Sprintlink traffic through your t1 and you'd hurt a reasonable chunk of the Internet. Both Sprintlink and UUNET do things to prevent you from doing this, but you should always be paranoid when dealing with BGP.)

Again, the "deny .*" rule is useless here, except as a safety precaution, since the router would insert that rule anyway (remember, there's an implicit "deny .*" at the end of every Cisco filter list).

A quick note: For those playing with BGP confederations on your own (a topic we'll talk about in a future document) note that your "permit internal routes only" filter might have to look something different ("permit ^$" will no longer be enough) - something like: "ip as-path access-list 30 permit ^(\([0-9 ]*\))*$". Or you'll be using BGP communities instead of AS-PATH filtering to control which routes you redistribute Everyone else please ignore this paragraph, unless you want to try to parse the regexp above as an exercise.

For Examples 4 and 5, please consult Fig 4 for a list of common ASNs you'll see when examining routes. To find out who "owns" an ASN (funny concept - owning a 16-bit integer), issue a WHOIS query on "ASN NNN", where NNN is the ASN. Note: You may actually need to put quotes around the "ASN NNN", especially if you're doing the whois query from a command line.

--------------------------------------------------------

3561  MCI
1239  Sprintlink (Sprintlink also uses other ASNs, but 1239 will always
      appear somewhere in the AS-PATH when looking at Sprintlink
      routes from some other provider)
701   UUNET
174   PSI
1673  ANS (the old ANS ASN, 690, should be retired by now)
1     BBN
4200  AGIS (the old Net99 ASN, 3830, should be retired by now)
4969  Net Access (which will appear in the examples)

-----------------------------------------------
Example 4:
ip as-path access-list 20 permit _1_
ip as-path access-list 20 permit _701_
ip as-path access-list 20 permit _174_
ip as-path access-list 20 permit _1673_
ip as-path access-list 20 permit _4200_
ip as-path access-list 20 deny .*

"_1_" would match "1"; "3561 1 6000"; and "3561 1" - but not "701". (ASN 1 is used by BBN, which has a bit of history in the Internet...)

So - this as-path access list permits, in order, BBN, UUNET, PSI, ANS, and AGIS routes, and denies all other routes. If you had a Cisco 2501, you might want to do this to accept some routes from one of your providers in an attempt to load-balance traffic a certain way (perhaps you've noticed that provider B gets better BBN connectivity than provider A...

Example 5:

ip as-path access-list 20 deny _3561_
ip as-path access-list 20 deny _1239_
ip as-path access-list 20 permit .*

This will fill up a 2501 with absolutely all of the routes it can take and still function well. It used to be that all routes on the 'net fit in a 2501 with 16mb - and that the 2501 could still function. Then, the routes would fit in but the 2501 didn't have enough CPU. Now, all of the routes on the 'net except for MCI, Sprintlink, or both will fit in a 2501 and still let it function at at least a single t1's worth of throughput.

ENTERING, MODIFYING, AND DELETING as-path access-lists

So, as a security blanket, appending an explicit "deny .*" to a list ensures that you will at least not be able to modify an existing list's functionality.

Let's say you had:

ip as-path access-list 3 permit ^$

ip as-path access-list 3 permit _1239_

But if you had:

ip as-path access-list 3 permit ^$
ip as-path access-list 3 deny .*

ip as-path access-list 3 permit _1239_

So, to modify an existing access list, either:

BGP METRICS (ATTRIBUTES) AND ROUTE SELECTION: INTRODUCTION

There are, however, rules for how a Cisco will select the "best BGP" route when there are multiple BGP route possibilities of the same specificity.

It goes (basically):

For "competing" BGP routes, the most likely way the router's going to pick the best route (if you aren't playing games with weights) is by looking at the AS-PATH lengths.

BGP PATH SELECTION PROCESS ACCORDING TO CISCO

"BGP selects only one path as the best path.
When the path is selected, BGP puts the selected
path in its routing table and propagates the path to
its neighbors. BGP uses the following criteria,
in the order presented, to select a path
for a destination: 

1. If the path specifies a next hop that is inaccessible, drop the update.

2. Prefer the path with the largest weight.

3. If the weights are the same, prefer the path with the largest local
preference.

4. If the local preferences are the same, prefer the path that was
originated by BGP running on this router.

5. If no route was originated, prefer the route that has the shortest
AS_path.

6. If all paths have the same AS_path length, prefer the path with the
lowest origin type (where IGP is lower than EGP, and EGP is lower than
Incomplete).

7. If the origin codes are the same, prefer the path with the lowest MED
attribute.

8. If the paths have the same MED, prefer the external path over the
internal path.

9. If the paths are still the same, prefer the path through the closest
IGP neighbor.

10. Prefer the path with the lowest IP address, as specified by the BGP
router ID."

Fig 8: BGP attributes

For more info, see:

RFC 2042: Registering New BGP Attribute Types
RFC 1997: BGP Communities Attribute
RFC 1773: Experience with the BGP-4 protocol
RFC 1771: A Border Gateway Protocol 4 (BGP-4)

To get an RFC, go to: http://www.internic.net/rfc/rfcXXXX.txt

BGP ATTRIBUTE TYPES

Value Code            Possible Values
---- ---------------- ------------------------------
1  ORIGIN           0 (IGP); 1 (EGP); 2 (Incomplete)
                    This attribute specifies the origin of a route.
                    Straightforward except that "Incomplete" means
                    that the route got into BGP by redistribution from
                    an IGP.
2  AS_PATH          0-N 2-byte values
                    A list of the ASNs of all ASs the route has traversed.
3  NEXT_HOP         IP Address
                    The most critical attribute; where to send data destined
                    for this route.
4  MULTI_EXIT_DISC  0-2^32
                    A weight; designed to go outside and inside of an ASN.
5  LOCAL_PREF       0-2^32
                    A weight; not designed to go outside of an ASN.
6  ATOMIC_AGGREGATE TRUE/FALSE: If present, true; otherwise, false.
                    Present if this route was not the most specific one
                    known by the advertiser.  Dangerous stuff.
7  AGGREGATOR       {ASN,Ip address} pair.
                    Data to indicate who formed the route if the route
                    is an aggregate of smaller routes.
8  COMMUNITY        0-N 4-byte values ("communities")
                    To be covered in a future document.
9  ORIGINATOR_ID    Used for BGP Route Reflection
                    To be covered in a future document.
0  CLUSTER_LIST     Used for BGP Route Reflection
                    To be covered in a future document.

Briefly:

(Rule 2)

(Rule 3)

(Rules 2-3,5)

(Rule 6)

(Rules 7-8)

(Rule 9)

(Rule 10)

For further reading, see for more details.

We'll be talking about using these metrics in the near future. If you want to experiment in the mean-time, that document shows you how to set these metrics. Please experiment first on test or lab networks! If you've got proper filters in place, experimenting with these things won't affect the outside world - but it could make your customers very unhappy...

Another very big caution: BGP weights and local_prefs are very powerful. Realize that if you advertise routes for a customer that you hear via BGP, you could wind up preferring an external route for that customer if you set the BGP weight or local_pref too high (or at all) for external routes. The customer won't like this - if you prefer an external route for that customer, you're not going to advertise them to your transit providers any more, which will probably not please that customer...

EGP vs. IGP

Routers which route IP packets have to have an "IP routing table". In that table are one or more routes of a particular {starting point, length, metric}. This IP routing table gets filled with routes heard from various sources - or configured statically (in the router's configuration store). BGP routes migrate into the IP routing table only if:

Here's a brief outline of the "order of preference" for filling the IP routing table. The exact order can be found in the Cisco documentation.

One note, though: Since static routes are really considered an "IGP" routing mechanism, there are ways to get other IGP-learned routes (say, via OSPF) to be preferred over static routes, but again - if you don't play with weights, this shouldn't be a worry.

WHAT IS ROUTE FLAP AND WHY IS IT BAD?

But look at what happens when you withdraw that assertion. Your provider(s) must then also withdraw that assertion. And then their provider(s) and peer(s) must do the same. All in all, thousands of routers around the world now have to look at that route and decide if they have a next-best path in their BGP (or other routing) table, and insert it as the current best path in their IP routing table. This consumes many CPU-seconds on routers that are sometimes very busy.

In fact, it was consuming so much CPU time a few years ago that Sean Doran of Sprintlink said "this must stop" and a few people came up with an idea (which Cisco implemented in record time) to "damp"(en) the "route flap"s. You'll hear people say "damp" and "dampen". There's no real consensus about which is the correct term.

What this means in practice today is that if your routes flap more than one or two complete up-down-up cycles, you will be dampened by many providers for at least an hour or so. So even if you're only "single-homed", you will be dampened if your provider withdraws your routes every time your t1 flips up and down a few times because some Bell guy tripped over a wire.

So do not ask your upstream provider to announce you unless it makes a difference (the benefit of being multiply-announced outweighs the possible negative effects of being dampened due to instability in either your or your provider's network).

WHAT TO KEEP IN MIND WHEN CONFIGURING BGP

BGP AND PEERING

What we're talking about in this document is BGP and transit - getting global transit from upstream providers as opposed to peering, which is just mutual sharing of customer routes.

INTERNET CONNECTIVITY WITHOUT BGP

Basically, if you have any address space "inside" of your provider's larger "netblock" or "aggregate", you won't be advertised to the outside world specifically - your provider will just advertise their larger block. If you have any other networks (an old Class C; customers with address space; etc...) your provider will just statically announce those routes to the world and statically route them inside their network to your leased-line/ router interface(s).

With BGP, your provider gives you all of the routes they have (the easy part), and listens to your route announcements and then redistributes some or all of those to their peers and customers. This is the hard part (for them - just worry about understanding and configuring your end for now). The net difference is "just" that they may start advertising a more specific route (no mean task in a complicated network designed, as most networks are, to prevent the accidental "leaking" of more specific routes) or that the routes that they normally advertise for you under just their ASN will now have your ASN attached as well.

BGP AND THE SINGLE-HOMED

And you don't really need "full routes" so that you can "run defaultless" if you're single-homed. Since every packet destined for the Internet (as opposed to your internal network ) is going to go out the same router interface, it doesn't matter whether it's via one default route or via searching a list of 45,000 or more routes heard via BGP.

The only really valid reason is that you want to be able to have more control in advertising your routes. Of course, you'll have to argue around the flap argument even if you have your own provider-independent address space (if you're singly- connected to the 'net, why bother all of the routers in the world by telling them whether you're reachable or not currently) and the routing-table space argument (if you're in your provider's IP space or "aggregate announcement"), why pollute the routing tables with an extra few routes by announcing your routes more specifically?

You're on your own for the answers to these questions. If you think you have a good case, either talk to your current or potential provider, or perhaps send a question off to the inet-access list and see if anyone can help.

If you do want to configure BGP and are single-homed, follow the instructions on how to announce your networks (routes), and either filter all incoming routes - or accept them if you feel you really want to.

BGP AND THE MULTI-HOMED

So the most important thing about being multi-homed is the ability to have your routes advertised to your providers - and by them to their providers and peers (i.e. to "the rest of the Internet"). Doing this basic level of route advertisement is not hard. You just have to do it in a paranoid way.

If you screw up BGP routing you may get slapped down pretty hard. Screwups with BGP route advertisements can be felt all over the Internet. To repeat: Screwups with BGP route advertisements can be felt all over the Internet. If your provider is smart, they will also implement "filters" to prevent you from screwing them and the Internet up. But don't count on it.

If you were to announce a route that was more specific than, say, the otherwise-best route for Yahoo's web servers, you would black-hole Yahoo for a period of time. Needless to say, they would not be very happy with you. The solution is to do good filtering on your end - and for your provider to also do excellent filtering wherever possible.

Before you start playing with BGP, you might really want to wait and read the "Configuring a Cisco Router" document (also coming out in the next few months). If you do go ahead and are implementing BGP for the first time, get a friend or another provider to review your proposed configs for you before implementing them. And for a summary of BGP-related Cisco commands, see the BGP Cisco Commands sidebar.

MULTI-HOMING AND LOAD-BALANCING

We'll talk a bit about how you load-balance incoming and outgoing traffic to and from your network. Incoming traffic is controlled by how you announce your routes to the world (packets will flow into your network because someone out there heard and is using a route announcement). Outgoing traffic is controlled by the routes that you allow to flow into your border router(s) - and is thus much easier to control and tune.

HOW TO ANNOUNCE YOUR NETWORKS

There are many other ways, some of which we'll talk about in future document. The way we at Net Access do it is by redistributing from our IGP (IS-IS), through a filter list, into BGP. While we do run BGP inside our network, it's strictly to pass external route announcements through the various parts of our network - no internal routes are ever passed from one of our routers to another one of our routers with BGP. But when we first started speaking BGP, we set our routers up the way described below.

You'll always set "next-hop-self" on all peering sessions. See the sidebar on next-hop-self for an explanation.

The safest way to announce your routes with BGP is to configure everything statically. You can think of the process described below as turning networks into route announcements.

To do this:

For example, let's say you're routing the following networks (also called "netblocks" sometimes):

170.100.0.0/16 (a /16 has a netmask of 255.255.0.0) 192.204.44.0/24 (a /24 has a netmask of 255.255.255.0) 206.8.128.0/17 (a /17 has a netmask of 255.255.128.0) 207.126.0.0/18 (a /18 has a netmask of 255.255.192.0)

You'd first configure your router with:

int Loopback0
descr Loopback interface for routes to be nailed to.
ip route 170.100.0.0 255.255.0.0 Loopback0 10
ip route 192.204.44.0 255.255.255.0 Loopback0 10
ip route 206.8.128.0 255.255.128.0 Loopback0 10
ip route 207.126.0.0 255.255.192.0 Loopback0 10

Then:

ip as-path access-list 2 deny .*
ip as-path access-list 3 permit ^$
ip as-path access-list 3 deny .*

router bgp 64512
 network 170.100.0.0 mask 255.255.0.0
 network 192.204.44.0 mask 255.255.255.0
 network 206.8.128.0 mask 255.255.128.0
 network 207.126.0.0 mask 255.255.192.0
 neighbor  remote-as 
 neighbor  next-hop-self
 neighbor  filter-list 3 out
 neighbor  filter-list 2 in

Explanation:

This method "statically nails down" the route announcements being advertised with the "network" statements. In order to nail them down, there must be: (1) Underlying static routes with the same netmask as each route being advertised with a network statement; and (2) Those underlying static routes must not go away. The purpose of the Loopback0 routes is to ensure that even if an existing primary route which matches the netmask of the route being announced (and this is often not the case) goes away, the Loopback0 route (with a weight of 10, which means it's only a "backup" route to any route without a weight at the end) will kick in and keep the BGP route advertisement stable. (Loopback0 routes always stay installed since there's no physical interface to go down and cause the route to be withdrawn - the interface Loopback0 will always be up, so the routes pointed to them will always be installed.)

This example uses a "deny everything" incoming filter, so it will only announce routes - it won't accept any. If you want to accept all incoming routes, replace the "filter-list 2 in" with "filter-list 1 in". Actually, you could just not specify an "inbound as-path filter" - and the effect would be the same - but it's better by far to be explicit about these things.

To add more peers, just create another similar neighbor statement. Ciscos give you 30 seconds to finish typing the neighbor statement before they start trying to establish the session. It is critical that you get those "neighbor somebody filter-list xxx .." statements in there by then. The best way by far to do it is to either cut and paste or tftp in a complete neighbor statement to the router.

Here's an example of a completely filled-in bgp clause, based on the example above (note that the 64512 is a fictitious IP address).

router bgp 64512
network 170.100.0.0 mask 255.255.0.0
network 192.204.44.0 mask 255.255.255.0
network 206.8.128.0 mask 255.255.128.0
network 207.126.0.0 mask 255.255.192.0
neighbor 207.106.127.45 remote-as 4969
neighbor 207.106.127.45 next-hop-self
neighbor 207.106.127.45 filter-list 3 out
neighbor 207.106.127.45 filter-list 2 in
neighbor 137.10.10.121 remote-as 701
neighbor 137.10.10.121 next-hop-self
neighbor 137.10.10.121 filter-list 3 out
neighbor 137.10.10.121 filter-list 2 in

BEING ADVERTISED BY MULTIPLE PROVIDERS WITHOUT PI-SPACE

Let's say you are using 207.106.96.0/20. Your provider (let's call him oldprovider) has 207.106.0.0/16. So oldprovider announces only 207.106.0.0/16 to the world. There is no advertisement for 207.106.96.0/20 in this case - any packet destined to 207.106.96.0/20 will be picked up by the less specific (more general) route 207.106.0.0./16.

Now you want to multi-home. So you buy a T1 from newprovider. You set up BGP with both oldprovider and newprovider. Suddenly, the world sees two routes for you:

207.106.0.0/16, advertised by oldprovider; and 207.106.96.0/20, advertised by newprovider.

Remember, the most specific route always wins, so newprovider will wind up carry almost all, if not all, of your incoming traffic! In fact, certain parts of oldprovider's network may actually prefer newprovider's t1 to get to you!

The problem is that most large-ish providers use something called "aggregate-address statements" - and they certainly have some sort of filter to keep the more specific routes floating around inside of their networks from being advertised to the world. Remember, the world only wants to hear about 207.106.0.0/16 if the little, more specific routes inside of 207.106.0.0 are not multi-homed.

So what does oldprovider have to do? Blow holes in their "filter". One way or another, it's going to take modifications in oldprovider's 'border' routers to make incoming load-balancing work properly for you - and oldprovider may not want to do this. Basically, everywhere that oldprovider peers with anyone else (and this is usually at least 5-10 places), they have to modify their aggregation statements or other filters to "allow" your more specific route announcement to pass through.

This is why it's important to choose a primary provider based on how cooperative they'll be when you want to multi-home.

CONTROLLING OUTGOING DATA FLOW: "FULL ROUTING"

There are a couple of reasons. First, each provider obviously knows best the way to get to its customers. Meaning, if you're multi-homed to Sprintlink and UUNET, you always want to send data to Sprintlink customers out your Sprintlink T1 and data to UUNET customers out your UUNET T1. Second, though AS-PATH length is a pretty poor selection tool, it's what we've got right now - and it does bear some relation to an indicator of how "close" a given provider is to some other provider.

So filling your router with routes from all of your upstream providers means that, for routes of the same specificity, AS-PATH length will decide which one actually gets used. See Fig 7 for examples and explanation.

CONTROLLING OUTGOING DATA FLOW: "PARTIAL ROUTING": "CUSTOMER ROUTES ONLY"

The minimum set of "less than full" routes you'll want to take is customer routes from each provider (from each provider, get only the routes for them and their customers). This is a problem if your providers include Sprintlink and MCI, however, since Sprintlink and MCI customer routes together are such a large percentage of "full routes" that you can't really put Sprintlink and MCI routes in Cisco 2501s or 4000s either. You should, however, be able to put Sprintlink and any other few sets of customer routes or MCI and any other few sets in even a 2501 or 4000.

The problem is getting just customer routes (also called "peering routes"). You can tell your providers to only send you customer routes - and most providers that do a significant amount of BGP can do this pretty easily - but if any one of your providers screws up (changes a filter list slowly, for example) then they may blast more than enough routes at you to "melt your router". Unfortunately, when many brands of routers (Ciscos included) run out of memory, they don't just shut down BGP routing - or crash and restart. Ciscos, in particular, do not handle running out of memory gracefully at all, and will gleefully consume so much memory with routing data that basic command functionality gets trashed and someone needs to physically power cycle the router.

SO WHAT'S TO BE DONE?

(Ciscos use ! at the beginning of a line to denote a comment line.)

! Filter everything but Sprintlink (ASN 1239) from Sprintlink
ip as-path access-list 40 deny _3561_
ip as-path access-list 40 deny _701_
ip as-path access-list 40 deny _1673_
ip as-path access-list 40 deny _174_
ip as-path access-list 40 deny _1_
ip as-path access-list 40 deny _4200_
ip as-path access-list 40 permit .*
! Filter everything but UUNET (ASN 701) from UUNET
ip as-path access-list 41 deny _3561_
ip as-path access-list 41 deny _1239_
ip as-path access-list 41 deny _1673_
ip as-path access-list 41 deny _174_
ip as-path access-list 41 deny _1_
ip as-path access-list 41 deny _4200_
ip as-path access-list 41 permit .*
! Filter the major providers from Net Access
ip as-path access-list 42 deny _3561_
ip as-path access-list 42 deny _1239_
ip as-path access-list 42 deny _701_
ip as-path access-list 42 deny _1673_
ip as-path access-list 42 deny _174_
ip as-path access-list 42 deny _1_
ip as-path access-list 42 deny _4200_
ip as-path access-list 42 permit .*

router bgp 64512
 
 neighbor  remote-as 1239
 neighbor  next-hop-self
 neighbor  filter-list 3 out
 neighbor  filter-list 40 in
 neighbor  remote-as 701
 neighbor  next-hop-self
 neighbor  filter-list 3 out
 neighbor  filter-list 41 in
 neighbor  remote-as 4969
 neighbor  filter-list 3 out
 neighbor  filter-list 42 out

That will ensure that even if Sprintlink, UUNET, or Net Access screw up and blow you all of the routes they know about, you'll still take their customer routes but won't take the vast majority of other routes from them. (Sprintlink, MCI, UUNET, ANS, PSI, BBN, and AGIS) make up the vast majority of routes - well over 80-85% of the routes out there.

Note: If you're a Sprintlink customer, you'll probably be peering with AS 179x - or at least some ASN other than 1239. Sprintlink uses ASNs for each major POP (as do many other providers) - but unlike other providers, these ASNs are visible to the outside world. Any non-Sprintlink customer route, though (any route from the outside world), will still have the ASN 1239 (which is Sprintlink's "peering" ASN) in the AS-PATH, though. The bottom line is that instead of below you'll have whatever ASN Sprintlink actually has you peer with.

AS-PATH PADDING

AS-PATH padding is probably the most widely-used BGP tuning method, and we'll go into it in more detail next month.

Basically, if you make sure not to set weights or local_prefs, AS-PATH length is going to decide which of multiple BGP routes of the same specificity will be preferred. So if you want to make one path preferred or another one not preferred, you can "pad" the AS-PATH with extra ASNs to make one path look longer than another. This is done with route-maps, which we'll talk more about next month.

QUESTIONS AND COMMENTS

THANKS TO

Thanks to Alexis Rosen at Panix (alexis@panix.com), who sent me some last-minute suggestions for clarification and pointed out an ugly factual error. Thanks to John Hawkinson (jhawk@panix.com) of BBN, who told me about something new called BGP in 1993 at a Science Fiction convention in the DC area. Thanks to Dave Siegel (dsiegel@rtd.net) who's shared his BGP experience with others since 1995. And thanks to Alec Peterson (ahp@hilander.com) for reviewing this document - and who explored some of the more advanced BGP features (oh, the joy of route-maps) using my network when I didn't have the time.

Sidebar on next-hop-self

Ciscos keep the originating address of a route intact in the next-hop field when they pass it from eBGP peer to eBGP peer. (And ditto for iBGP, but we're talking about eBGP here). It turns out that this behavior is sometimes useful in large networks where there's an IGP running to tell every router which way to send a packet that says it came from 192.41.177.x (some other provider's MAE-East router); 192.157.69.x (some other provider's Pennsauken router); etc...

But this is really subtle and can screw you up big-time. In the best case you'll piss someone off (if you forget to set "next-hop-self" in an exchange-point peering environment. In the worst case you'll cause routing loops for yourself (examples of this will be given when we talk more about IGPs).

Setting next-hop-self causes a Cisco to override the originating address of a route and stamp instead its own address as the "next-hop" part of the route.

Remember that the critical parts of a route are: What the base IP address is; how big the route is (the specificity or netmask); and what destination (next-hop) to use to send data to the IP space represented by the route.

We'll use an exchange point environment to illustrate next-hop-self. Refer to the figure (XXX) below. When AS 4969 advertises 250.20.0.0/16 to AS 64500, AS 4969 sets next-hop-self, so the next-hop is 192.41.177.87 (AS 4969's mae-east IP address).

Now, AS 64500 advertises it to AS 64600 (see the top diagram) without next-hop-self. When AS 64600 processes the route and installs it into the IP routing table, the next-hop used will be 192.41.177.87.

But AS 64600 doesn't peer with AS 4969 - yet it's going to send data to a route advertised by AS 4969 - right to AS 4969's router. People generally do not like this. In this case, AS 4969 might discover this "behavior" by running a few careful probes of other routers at mae-east. AS 4969 would then look to see how it hears AS 64600 (who is announcing AS 64600 to AS 4969) and see if they're the culprits. If AS 4969 really wants to, it can find out who the culprit is by passing a bogus route or two to each peer in turn, and see when AS 64600's router starts using the bogus route.

The solution is for 64500 to use next-hop-self as well (see the bottom diagram). In this case, the route as heard by 64600 has 192.41.177.NNN (AS 64500's mae-east IP address) in the next-hop field - though the AS-PATH and certain other fields still show that AS 4969 is the origin of the route. So when AS 64600 wants to send data to AS 4969 based on this route it'll "bounce the traffic off of" AS 64500's router. Some people don't even like this (since it's a form of providing service to downstream customers over the "shared medium" of the exchange-point switches), but it's not going to be as strenuously objected to as not using next-hop-self.

Sidebar on Outgoing Data Flow Control Without BGP

Any packet not destined to the inside of the ISP's network will then hit the "wildcard", or "default" route, and be sent out the router interface towards the provider(s).

There are a few ways you can do this.

Outgoing Data Flow: Option 1

Option 1 is to default to one provider and install a "backup default" to your other provider. On a Cisco, this is done with:

ip route 0.0.0.0 0.0.0.0 Serial0
ip route 0.0.0.0 0.0.0.0 Serial1 10

If you do it this way, the route with a lower weight will be around when Serial0 is up. If Serial0 goes down for some reason (actually, if the "line protocol" on Serial0 goes down), the route will be invalidated and will go away, so the Cisco will look for the next-best route, which will be the route through Serial1. Even though it has a lower weight, it's the only valid route left to consider, so it'll "win".

Outgoing Data Flow: Option 2

Option 2 is to default equally to both providers. However, there's a catch. If you just do:

ip route 0.0.0.0 0.0.0.0 Serial0
ip route 0.0.0.0 0.0.0.0 Serial1

The fix is easy, however:

int Serial0
 ip route-cache
int Serial1
 ip route-cache

And actually, many Ciscos come pre-configured with "ip route-cache" set on all of the interfaces - but even so, it doesn't hurt to be explicit.

If you do this, the Cisco will keep a cache of all destinations you're sending packets to, and will "lock in" each destination to one specific interface. In general, this method leads to decent load-balancing (in the 40/60 to 50/50 split range). The worst case in this scenario is not IP degradation, but poor use of your additional bandwidth (which can, of course, lead to IP degradation if you need your second outgoing pipe because your first has a tendency to get full). Anyway, this kind of load-balancing works pretty well and is what people use when they can't accept "full BGP routes" from multiple providers.

TO BE DONE