Часть 4. Underlay. BGP
Цель
Настроить BGP для Underlay сети
Ожидаемый результат
Настроен BGP в Underlay сети и IP-связанность между всему сетевыми устройствами.
В документации зафиксирован план работ, адресное пространство, схема сети и конфигурация устройств.
Подтверждена IP-связанность между устройствами.
Планирование AS
Под катом планирование номеров AS
Disclaimer
После сложной рабочей недели планирование AS пришлось делать с помощью чат-бота
Предлагается следующая схема использования нумерации AS:
Используем 32-битную нумерацию.
Используем 32-битные частные ASNs из диапазона 4200000000–4294967294.
Первые 16 бит отводятся для обозначения страны, остальные для использования внутри страны.
Кодируем каждую страну символом:
Базовый AS номер для каждой страны вычисляется по формуле:
Базовый AS = 4200000000 + (Код страны × 65536)
Пример для Норвегии:
AS = 4200000000 + (2 × 65536) = 4200131072
Структура внутри страны:
[Код города (8 бит)][Код региона/ДЦ (8 бит)]
Пример для Норвегии (NO)
Базовый AS для города вычисляется следующим образом:
Базовый AS города = Базовый AS страны + (Код города × 256)
Пример для Осло (OSL)
AS = 4200131072 + (1 × 256) = 4200131328
Расчет распределения AS внутри города по датацентрам.
Для DC1 ("Мидгард")
AS = 4200131328 + 1 = 4200131329
Spine-коммутаторы: AS 4200131329
Leaf-коммутаторы: диапазон 4200131331–4200131583 для уникальных ASNs.
Заметка.
Распределение AS нужно пересмотреть в пределах Города-ДЦ. Пока используем так.
Для Spine-коммутаторов внутри DC1 ("Мидгард") используем следующую AS: 4200131329
Для каждого Leaf-коммутатора используем уникальный номер AS:
no-osl-dc1-f1-r03k01-lf01 - 4200131331
no-osl-dc1-f1-r03k02-lf01 - 4200131332
no-osl-dc1-f1-r03k03-lf01 - 4200131333
То же самое, но табличкой:
no-osl-dc1-f1-r01k01-spn01
Spine01
4200131329
no-osl-dc1-f1-r02k01-spn01
Spine02
4200131329
no-osl-dc1-f1-r03k01-lf01
Leaf01
4200131331
no-osl-dc1-f1-r03k02-lf01
Leaf02
4200131332
no-osl-dc1-f1-r03k03-lf01
Leaf03
4200131333
На картинке выглядит так:

Достижение результата
В результате поисков путей реализации лабораторный Netbox был дополнен двумя, на мой взгляд, полезными плагинами:
NetBox BGP Plugin https://github.com/netbox-community/netbox-bgp/ - документирование BGP.
NextBox UI Plugin https://github.com/iDebugAll/nextbox-ui-plugin - визуализация топологий.
Итак, мы имеем установленный и настроенный Netbox с плагином "NetBox BGP Plugin". Далее, нам нужно заполнить информацию о BGP, а именно:
Создать в разделе
IPAM -> Aggregates -> RIRs
хотя бы один RIR

Создать в разделе
IPAM -> ASNS -> ASNs
автономные системы согласно таблички выше и соотнести их к RIR:

Заполнить информацию о BGP сессиях в разделе
Plugins ->BGP -> Sessions
:

Примечание
Для массового добавления нужно использовать импорт.
Высокий риск допустить ошибку при ручном заполнении.
Применяем новый шаблон для генерации конфигурации устройств и BGP netbox_bgp_template.
netbox_bgp_template.jinja2
hostname {{ device.name }}
!
{%- block content %}
management api http-commands
no shutdown
!
vrf default
no shutdown
!
vrf {{ vrfs.mgmt }}
no shutdown
{%- endblock %}
!
ip routing
no ip routing vrf {{ vrfs.mgmt }}
!
ip route vrf {{ vrfs.mgmt }} 0.0.0.0/0 {{ mgmt_default_gw }}
!
{%- for key, value in stp_mode.items() %}
spanning-tree mode {{ value }}
!
{%- endfor %}
{%- for key, value in vrfs.items() %}
vrf instance {{ value }}
!
{%- endfor %}
{%- for interface in device.interfaces.all() %}
{%- if interface.name.startswith('Ethernet') %}
interface {{ interface.name }}
{%- for ip in interface.ip_addresses.all() %}
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address {{ ip.address }}
{%- endfor %}
{%- if not interface.enabled %}
shutdown
{%- else %}
no shutdown
{%- endif %}
{%- if interface.description %}
description {{ interface.description }}
{%- endif %}
!
{%- elif interface.name.startswith('Loopback') %}
interface {{ interface.name }}
{%- for ip in interface.ip_addresses.all() %}
ip address {{ ip.address }}
{%- endfor %}
{%- if interface.description %}
description {{ interface.description }}
{%- endif %}
!
{%- elif interface.name.startswith('Management') %}
interface {{ interface.name }}
{%- for ip in interface.ip_addresses.all() %}
ip address {{ ip.address }}
vrf {{ vrfs.mgmt }}
{%- endfor %}
{%- if interface.description %}
description {{ interface.description }}
{%- endif %}
!
{%- endif %}
{%- endfor %}
!
! Данный блок отвечает за генерацию конфигурации для BGP из Netbox
{%- for local_asn, sessions in device.bgpsession_set.all() | groupby('local_as.asn') %}
router bgp {{ local_asn }}
maximum-paths 2
{%- for session in sessions %}
neighbor {{ session.remote_address.address.ip }} remote-as {{ session.remote_as.asn }}
neighbor {{ session.remote_address.address.ip }} bfd
{%- endfor %}
redistribute connected
{%- endfor %}
!
end
Применяем Rendered Config к устройствам (все еще вручную)
no-osl-dc1-f1-r01k01-spn01.conf
hostname no-osl-dc1-f1-r01k01-spn01
!
management api http-commands
no shutdown
!
vrf default
no shutdown
!
vrf mgmt
no shutdown
!
ip routing
no ip routing vrf mgmt
!
ip route vrf mgmt 0.0.0.0/0 172.16.108.1
!
spanning-tree mode mstp
!
vrf instance mgmt
!
interface Ethernet1
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.0/31
no shutdown
!
interface Ethernet2
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.2/31
no shutdown
!
interface Ethernet3
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.4/31
no shutdown
!
interface Ethernet4
shutdown
!
interface Ethernet5
shutdown
!
interface Ethernet6
shutdown
!
interface Ethernet7
shutdown
!
interface Ethernet8
shutdown
!
interface Ethernet9
shutdown
!
interface Ethernet10
shutdown
!
interface Ethernet11
shutdown
!
interface Ethernet12
shutdown
!
interface Ethernet13
shutdown
!
interface Ethernet14
shutdown
!
interface Ethernet15
shutdown
!
interface Ethernet16
shutdown
!
interface Loopback0
ip address 10.16.0.1/32
description Loopback for RE
!
interface Management1
ip address 172.16.108.101/24
vrf mgmt
!
!
router bgp 4200131329
maximum-paths 2
neighbor 10.16.2.1 remote-as 4200131331
neighbor 10.16.2.1 bfd
neighbor 10.16.2.3 remote-as 4200131332
neighbor 10.16.2.3 bfd
neighbor 10.16.2.5 remote-as 4200131333
neighbor 10.16.2.5 bfd
redistribute connected
!
end
no-osl-dc1-f1-r02k01-spn01.conf
hostname no-osl-dc1-f1-r02k01-spn01
!
management api http-commands
no shutdown
!
vrf default
no shutdown
!
vrf mgmt
no shutdown
!
ip routing
no ip routing vrf mgmt
!
ip route vrf mgmt 0.0.0.0/0 172.16.108.1
!
spanning-tree mode mstp
!
vrf instance mgmt
!
interface Ethernet1
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.6/31
no shutdown
!
interface Ethernet2
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.8/31
no shutdown
!
interface Ethernet3
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.10/31
no shutdown
!
interface Ethernet4
no shutdown
!
interface Ethernet5
no shutdown
!
interface Ethernet6
no shutdown
!
interface Ethernet7
no shutdown
!
interface Ethernet8
no shutdown
!
interface Ethernet9
no shutdown
!
interface Ethernet10
no shutdown
!
interface Ethernet11
no shutdown
!
interface Ethernet12
no shutdown
!
interface Ethernet13
no shutdown
!
interface Ethernet14
no shutdown
!
interface Ethernet15
no shutdown
!
interface Ethernet16
no shutdown
!
interface Loopback0
ip address 10.16.0.2/32
description Loopback for RE
!
interface Management1
ip address 172.16.108.102/24
vrf mgmt
!
!
router bgp 4200131329
maximum-paths 2
neighbor 10.16.2.7 remote-as 4200131331
neighbor 10.16.2.7 bfd
neighbor 10.16.2.9 remote-as 4200131332
neighbor 10.16.2.9 bfd
neighbor 10.16.2.11 remote-as 4200131333
neighbor 10.16.2.11 bfd
redistribute connected
!
end
no-osl-dc1-f1-r03k01-lf01.conf
hostname no-osl-dc1-f1-r03k01-lf01
!
management api http-commands
no shutdown
!
vrf default
no shutdown
!
vrf mgmt
no shutdown
!
ip routing
no ip routing vrf mgmt
!
ip route vrf mgmt 0.0.0.0/0 172.16.108.1
!
spanning-tree mode mstp
!
vrf instance mgmt
!
interface Ethernet1
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.1/31
no shutdown
!
interface Ethernet2
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.7/31
no shutdown
!
interface Ethernet3
no shutdown
!
interface Ethernet4
no shutdown
!
interface Ethernet5
no shutdown
!
interface Ethernet6
no shutdown
!
interface Ethernet7
no shutdown
!
interface Ethernet8
no shutdown
!
interface Ethernet9
no shutdown
!
interface Ethernet10
no shutdown
!
interface Ethernet11
no shutdown
!
interface Ethernet12
no shutdown
!
interface Ethernet13
no shutdown
!
interface Ethernet14
no shutdown
!
interface Ethernet15
no shutdown
!
interface Ethernet16
no shutdown
!
interface Loopback0
ip address 10.16.1.1/32
description Loopback for RE
!
interface Loopback10
ip address 10.16.4.1/32
description Loopback for VTEP
!
interface Management1
ip address 172.16.108.111/24
vrf mgmt
!
!
router bgp 4200131331
maximum-paths 2
neighbor 10.16.2.0 remote-as 4200131329
neighbor 10.16.2.0 bfd
neighbor 10.16.2.6 remote-as 4200131329
neighbor 10.16.2.6 bfd
redistribute connected
!
end
no-osl-dc1-f1-r03k02-lf01.conf
hostname no-osl-dc1-f1-r03k02-lf01
!
management api http-commands
no shutdown
!
vrf default
no shutdown
!
vrf mgmt
no shutdown
!
ip routing
no ip routing vrf mgmt
!
ip route vrf mgmt 0.0.0.0/0 172.16.108.1
!
spanning-tree mode mstp
!
vrf instance mgmt
!
interface Ethernet1
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.3/31
no shutdown
!
interface Ethernet2
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.9/31
no shutdown
!
interface Ethernet3
no shutdown
!
interface Ethernet4
no shutdown
!
interface Ethernet5
no shutdown
!
interface Ethernet6
no shutdown
!
interface Ethernet7
no shutdown
!
interface Ethernet8
no shutdown
!
interface Ethernet9
no shutdown
!
interface Ethernet10
no shutdown
!
interface Ethernet11
no shutdown
!
interface Ethernet12
no shutdown
!
interface Ethernet13
no shutdown
!
interface Ethernet14
no shutdown
!
interface Ethernet15
no shutdown
!
interface Ethernet16
no shutdown
!
interface Loopback0
ip address 10.16.1.2/32
description Loopback for RE
!
interface Loopback10
ip address 10.16.4.2/32
description Loopback for VTEP
!
interface Management1
ip address 172.16.108.112/24
vrf mgmt
!
!
router bgp 4200131332
maximum-paths 2
neighbor 10.16.2.2 remote-as 4200131329
neighbor 10.16.2.2 bfd
neighbor 10.16.2.8 remote-as 4200131329
neighbor 10.16.2.8 bfd
redistribute connected
!
end
no-osl-dc1-f1-r03k03-lf01.conf
hostname no-osl-dc1-f1-r03k03-lf01
!
management api http-commands
no shutdown
!
vrf default
no shutdown
!
vrf mgmt
no shutdown
!
ip routing
no ip routing vrf mgmt
!
ip route vrf mgmt 0.0.0.0/0 172.16.108.1
!
spanning-tree mode mstp
!
vrf instance mgmt
!
interface Ethernet1
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.5/31
no shutdown
!
interface Ethernet2
no switchport
bfd interval 200 min-rx 200 multiplier 3
ip address 10.16.2.11/31
no shutdown
!
interface Ethernet3
no shutdown
!
interface Ethernet4
no shutdown
!
interface Ethernet5
no shutdown
!
interface Ethernet6
no shutdown
!
interface Ethernet7
no shutdown
!
interface Ethernet8
no shutdown
!
interface Ethernet9
no shutdown
!
interface Ethernet10
no shutdown
!
interface Ethernet11
no shutdown
!
interface Ethernet12
no shutdown
!
interface Ethernet13
no shutdown
!
interface Ethernet14
no shutdown
!
interface Ethernet15
no shutdown
!
interface Ethernet16
no shutdown
!
interface Loopback0
ip address 10.16.1.3/32
description Loopback for RE
!
interface Loopback10
ip address 10.16.4.3/32
description Loopback for VTEP
!
interface Management1
ip address 172.16.108.113/24
vrf mgmt
!
!
router bgp 4200131333
maximum-paths 2
neighbor 10.16.2.4 remote-as 4200131329
neighbor 10.16.2.4 bfd
neighbor 10.16.2.10 remote-as 4200131329
neighbor 10.16.2.10 bfd
redistribute connected
!
end
Результат настройки BGP
no-osl-dc1-f1-r03k01-lf01: sh ip route bgp
VRF: default
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route
B E 10.16.0.1/32 [200/0] via 10.16.2.0, Ethernet1
B E 10.16.0.2/32 [200/0] via 10.16.2.6, Ethernet2
B E 10.16.1.2/32 [200/0] via 10.16.2.0, Ethernet1
via 10.16.2.6, Ethernet2
B E 10.16.1.3/32 [200/0] via 10.16.2.0, Ethernet1
via 10.16.2.6, Ethernet2
B E 10.16.2.2/31 [200/0] via 10.16.2.0, Ethernet1
B E 10.16.2.4/31 [200/0] via 10.16.2.0, Ethernet1
B E 10.16.2.8/31 [200/0] via 10.16.2.6, Ethernet2
B E 10.16.2.10/31 [200/0] via 10.16.2.6, Ethernet2
B E 10.16.4.2/32 [200/0] via 10.16.2.0, Ethernet1
via 10.16.2.6, Ethernet2
B E 10.16.4.3/32 [200/0] via 10.16.2.0, Ethernet1
via 10.16.2.6, Ethernet2
no-osl-dc1-f1-r03k02-lf01: sh ip route bgp
VRF: default
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route
B E 10.16.0.1/32 [200/0] via 10.16.2.2, Ethernet1
B E 10.16.0.2/32 [200/0] via 10.16.2.8, Ethernet2
B E 10.16.1.1/32 [200/0] via 10.16.2.2, Ethernet1
via 10.16.2.8, Ethernet2
B E 10.16.1.3/32 [200/0] via 10.16.2.2, Ethernet1
via 10.16.2.8, Ethernet2
B E 10.16.2.0/31 [200/0] via 10.16.2.2, Ethernet1
B E 10.16.2.4/31 [200/0] via 10.16.2.2, Ethernet1
B E 10.16.2.6/31 [200/0] via 10.16.2.8, Ethernet2
B E 10.16.2.10/31 [200/0] via 10.16.2.8, Ethernet2
B E 10.16.4.1/32 [200/0] via 10.16.2.2, Ethernet1
via 10.16.2.8, Ethernet2
B E 10.16.4.3/32 [200/0] via 10.16.2.2, Ethernet1
via 10.16.2.8, Ethernet2
no-osl-dc1-f1-r03k03-lf01: sh ip route bgp
VRF: default
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route
Gateway of last resort is not set
B E 10.16.0.1/32 [200/0] via 10.16.2.4, Ethernet1
B E 10.16.0.2/32 [200/0] via 10.16.2.10, Ethernet2
B E 10.16.1.1/32 [200/0] via 10.16.2.4, Ethernet1
via 10.16.2.10, Ethernet2
B E 10.16.1.2/32 [200/0] via 10.16.2.4, Ethernet1
via 10.16.2.10, Ethernet2
C 10.16.1.3/32 is directly connected, Loopback0
B E 10.16.2.0/31 [200/0] via 10.16.2.4, Ethernet1
B E 10.16.2.2/31 [200/0] via 10.16.2.4, Ethernet1
C 10.16.2.4/31 is directly connected, Ethernet1
B E 10.16.2.6/31 [200/0] via 10.16.2.10, Ethernet2
B E 10.16.2.8/31 [200/0] via 10.16.2.10, Ethernet2
C 10.16.2.10/31 is directly connected, Ethernet2
B E 10.16.4.1/32 [200/0] via 10.16.2.4, Ethernet1
via 10.16.2.10, Ethernet2
B E 10.16.4.2/32 [200/0] via 10.16.2.4, Ethernet1
via 10.16.2.10, Ethernet2
C 10.16.4.3/32 is directly connected, Loopback10
Проверяем доступность loopback интерфейсов
ping
no-osl-dc1-f1-r03k01-lf01#ping 10.16.1.2 source 10.16.1.1
PING 10.16.1.2 (10.16.1.2) from 10.16.1.1 : 72(100) bytes of data.
80 bytes from 10.16.1.2: icmp_seq=1 ttl=63 time=10.2 ms
80 bytes from 10.16.1.2: icmp_seq=2 ttl=63 time=6.67 ms
80 bytes from 10.16.1.2: icmp_seq=3 ttl=63 time=6.87 ms
80 bytes from 10.16.1.2: icmp_seq=4 ttl=63 time=7.18 ms
80 bytes from 10.16.1.2: icmp_seq=5 ttl=63 time=6.53 ms
--- 10.16.1.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 38ms
rtt min/avg/max/mdev = 6.539/7.504/10.256/1.396 ms, ipg/ewma 9.632/8.831 ms
no-osl-dc1-f1-r03k01-lf01#ping 10.16.1.3 source 10.16.1.1
PING 10.16.1.3 (10.16.1.3) from 10.16.1.1 : 72(100) bytes of data.
80 bytes from 10.16.1.3: icmp_seq=1 ttl=63 time=8.93 ms
80 bytes from 10.16.1.3: icmp_seq=2 ttl=63 time=6.03 ms
80 bytes from 10.16.1.3: icmp_seq=3 ttl=63 time=6.54 ms
80 bytes from 10.16.1.3: icmp_seq=4 ttl=63 time=6.92 ms
80 bytes from 10.16.1.3: icmp_seq=5 ttl=63 time=6.67 ms
--- 10.16.1.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 34ms
rtt min/avg/max/mdev = 6.033/7.023/8.933/1.003 ms, ipg/ewma 8.546/7.960 ms
no-osl-dc1-f1-r03k01-lf01#ping 10.16.4.2 source 10.16.4.1
PING 10.16.4.2 (10.16.4.2) from 10.16.4.1 : 72(100) bytes of data.
80 bytes from 10.16.4.2: icmp_seq=1 ttl=63 time=8.34 ms
80 bytes from 10.16.4.2: icmp_seq=2 ttl=63 time=7.16 ms
80 bytes from 10.16.4.2: icmp_seq=3 ttl=63 time=11.9 ms
80 bytes from 10.16.4.2: icmp_seq=4 ttl=63 time=7.64 ms
80 bytes from 10.16.4.2: icmp_seq=5 ttl=63 time=9.38 ms
--- 10.16.4.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 37ms
rtt min/avg/max/mdev = 7.162/8.896/11.944/1.700 ms, ipg/ewma 9.480/8.645 ms
no-osl-dc1-f1-r03k01-lf01#ping 10.16.4.3 source 10.16.4.1
PING 10.16.4.3 (10.16.4.3) from 10.16.4.1 : 72(100) bytes of data.
80 bytes from 10.16.4.3: icmp_seq=1 ttl=63 time=11.0 ms
80 bytes from 10.16.4.3: icmp_seq=2 ttl=63 time=10.6 ms
80 bytes from 10.16.4.3: icmp_seq=3 ttl=63 time=9.23 ms
80 bytes from 10.16.4.3: icmp_seq=4 ttl=63 time=8.72 ms
80 bytes from 10.16.4.3: icmp_seq=5 ttl=63 time=11.5 ms
--- 10.16.4.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 48ms
rtt min/avg/max/mdev = 8.722/10.251/11.557/1.090 ms, ipg/ewma 12.173/10.661 ms
Last updated