Cisco Support Community Expert Series Webcast: Understanding Cisco ASR 9000 Series Aggregation Services Routers Platform Architecture and Packet Forwarding Troubleshooting Xander Thuijs, CCIE #6775 ISP/VoIP Principal Engineer, ASR9000 & IOS-XR 5/14/2013
© 2013 Cisco and/or its affiliates. All rights reserved.
1
• Today’s featured expert is Cisco Engineer “Expert” • Ask him and the team questions now about the ASR9000 and
IOS-XR
Expert’s photo
Xander Thuijs CCIE #6775 in ISP and VoIP
© 2013 Cisco and/or its affiliates. All rights reserved.
2
Members of the ASR9000 Escalation team present today:
• Aleksandar Vidakovic Sr Technical Leader, Spain
• Sadanande Phadke Technical Leader, Boston, MA
• Krishna Eranti SW Engineer, RTP, NC
© 2013 Cisco and/or its affiliates. All rights reserved.
3
Today’s presentation will include audience polling questions We encourage you to participate!
© 2013 Cisco and/or its affiliates. All rights reserved.
4
If you would like a copy of the presentation slides, click the PDF link in the chat box on the right or go to https://supportforums.cisco.com/community/netpro/networkinfrastructure/routing Or, https://supportforums.cisco.com/docs/DOC-32967
© 2013 Cisco and/or its affiliates. All rights reserved.
5
Everyone who joins today’s webcast will receive:
125 Cisco Preferred Access Points!
© 2013 Cisco and/or its affiliates. All rights reserved.
6
What is your level of experience with ASR9000? a) I am new to the ASR9000 and IOS-XR b) I have a good understanding of IOS-XR, but I am
relatively new to ASR9000 c) I have a good understanding of IOS-XR and I know how
to work with the system d) I am using ASR9000 already and I am pretty good with it
© 2013 Cisco and/or its affiliates. All rights reserved.
7
Use the Q&A panel to submit your questions. Experts will start responding those
© 2013 Cisco and/or its affiliates. All rights reserved.
8
ASR9000 Architecture
Xander Thuijs Principal Engineer Cisco’s Product Security Incident Response Team (PSIRT) Security Research and Operations
© 2013 Cisco and/or its affiliates. All rights reserved.
9
• Platform architecture • Linecard Architecture • Packet flow through the Linecard and Fabric • Feature order of operation • NPU Architecture • Scale • QOS implementation • FIB implementation
We will not be discussing the in depths of IOS-XR or configuration today (to be scheduled for a future session) © 2013 Cisco and/or its affiliates. All rights reserved.
10
Next-Generation SP Edge & Aggregation Optimized for high dense10G & 100G aggregation
IOS-XR Non-Stopping service
Video DNA
440G/Slot Longevity Green
Fully converged L2 & L3 service
© 2013 Cisco and/or its affiliates. All rights reserved.
ANA service management and provisioning
11
“At-a-Glance” Fully Distributed Architecture for High Performance and High Multidimensional Control Plane Scale § Data forwarding is fully distributed on the line cards § Control plane split among RSP and LC CPU (same type of CPU as RSP)
Line Card RSP CPU CPU
BITS/DTI
FIA FIC
§ L2 protocols, BFD, CFM, Netflow runs on the LC CPU for high scale
True Modular OS for HA and Operational Simplicity § Micro-kernel based, true modular OS
Switch Fabric
§ High availability and System stability § SW patch granularity for operational simplicity
© 2013 Cisco and/or its affiliates. All rights reserved.
Active-Active Switch Fabric
Guarantee “0” packet loss during RSP failover 12
Integrated cable management with cover System fan trays
Front-toback airflow
Side-to-back airflow RSP (0-1)
Line Card (0-3)
Line Card (0-3, 4-7)
RSP (0-1)
System fan trays Air draw
cable management
© 2013 Cisco and/or its affiliates. All rights reserved.
Three Modular Power Supplies
Six Modular Power Supplies 13
10 slots chassis (v1 power layout shown) Shelf 0 (Top)
LC Feed A
PS 0 LC PS 1
Feed A
PS 2
Feed B
Feed B
PS 3 PS 4
Power Distribution Bus
Feed A
LC RSP RSP LC LC
§ Single power zone, one distribution bus § All modules load share § AC power supplies are rates @ 3KW each § ‘A’ feed wired to top power shelf § ‘B’ feed wired to bottom power shelf
Fans Feed B
PS 5
Fans
Shelf 1 (bottom) 14
10 slots chassis (v1 power layout shown) Shelf 0 (Top)
LC Feed A
PS 0
Feed B
Feed B
Feed A Feed B
Feed A Feed B
Feed A Feed B
PS 1 PS 2
PS 3 PS 4
Power Distribution Bus
Feed A
LC
§ Single power zone, one distribution bus
LC
§ All modules load share
RSP RSP LC LC Fans
Feed A Feed B
PS 5
§ 2kW and 1.5kW supplies § Each power supply is wired to both ‘A’ and ‘B’ feed § Feed failure doubles draw on remaining feed § supply failure increases draw on remaining supplies
Fans
Shelf 1 (bottom) © 2013 Cisco and/or its affiliates. All rights reserved.
15
Current RSP2
RSP440
Processors
2 x 1.5GHz Freescale 8641D CPU
Intel x86 Jasper Forest 4 Core 2.27 GHz
RAM (user expandable)
4GB @133MHz SDR
6GB (RSP440-TR) and 12GB (RSP440-SE) version @1066MHz DDR3
Cache
L1: 32KB L2: 1MB
L1: 32KB per Core L2: 8MB shared
Primary persistent storage
4GB
16GB - SDD
Secondary persistent storage (HD/SSD)
30GB - HDD
16GB - SDD
USB 2.0 port
No
Yes
Acceleration / Security
No
Yes
HW assisted CPU queues
No
Yes
nV Cluster – EOBC ports
No
Yes, 2 x 1G/10G SFP+
Switch fabric bandwidth
184G/slot (with dual RSP)
440G/slot (with dual RSP)
© 2013 Cisco and/or its affiliates. All rights reserved.
8GB
RSP440 16
BITS
Clock
Time FPGA Timing Domain
4/8GB MEM
HDD CF card Mgt Eth
Ether Switch
CPU
Mgt Eth Console Aux
4G CF
Punt FPGA I/O FPGA
Alarm
NVRAM Front Panel © 2013 Cisco and/or its affiliates. All rights reserved.
Boot Flash CPU Complex
Fabric Interface
Arbitration
EOBC/ Internal GE switch
Arbitration
Crossbar Fabric ASIC
Crossbar Fabric ASIC
Fabric Complex 17
First-generation LC (Trident NP)
-L, -B, -E
A9K-40G
A9K-4T
A9K-8T/4
A9K-2T20G
A9K-8T
A9K-16T/8
-TR, -SE Second-generation LC (Typhoon NP)
A9K-24x10GE
A9K-36x10GE © 2013 Cisco and/or its affiliates. All rights reserved.
A9K-2x100GE
A9K-MOD80
A9K-MOD160
MPAs 20x1GE 2x10GE 4x10GE 1x40GE 2x40GE 18
40G Example: 4x10GE 2GB flash
XFP 3
XFP 2
4GB memory
CPU
10GE PHY
NPU 0
XFP 0
Crossbar Fabric ASIC
Bridge FPGA 0
10GE PHY
Arbitration
NPU 1
RSP0 GigE EOBC
XFP 1
Crossbar Fabric ASIC
Fabric Interface
10GE PHY
NPU 2
Crossbar Fabric ASIC
Bridge FPGA 1
Arbitration
10GE PHY
I/O daughter card © 2013 Cisco and/or its affiliates. All rights reserved.
Crossbar Fabric ASIC
NPU 3
RSP1 Network Clocking
via backplane 19
80G Line rate
Number of HW elements on LC doubles: - 8 NPUs, 2 Fabric Interfaces, 8 Fabric Channels
2GB flash
XFP 3
10GE PHY
XFP 7
10GE PHY
XFP 2
10GE PHY
XFP 6
10GE PHY
XFP 1
10GE PHY
XFP 5
10GE PHY
4GB memory
10GE PHY
XFP 4
10GE PHY
© 2013 Cisco and/or its affiliates. All rights reserved.
Crossbar Fabric ASIC
NPU 1 NPU 2
Raven FPGA 0
NPU 4
GigE EOBC Fabric Interface
NPU 5
NPU 7
Arbitration
Fabric Interface
NPU 3
NPU 6
XFP 0
CPU
NPU 0
Crossbar Fabric ASIC
Raven FPGA 1
RSP0 Crossbar Fabric ASIC Crossbar Fabric ASIC Arbitration
Network Clocking
RSP1 via backplane 20
forwarding “slice” physical interfaces
© 2013 Cisco and/or its affiliates. All rights reserved.
FIA
NP
physical interface NP s physical interface NP s physical interface NP s physical interface s physical interface s
replicate “slices” of components to improve performance
RSP 3 Switch Fabric
FIA FIA FIA FIA
NP NP
FIA
Fabric Complex
Typhoon LC CPU: Freescale Quad core P4040 21
3x10GE SFP + 3x10GE SFP +
Typhoon Typhoon 3x 10G
Typhoon
RSP Switch Fabric SFP3 +
3x10GE SFP + 3x10GE SFP + 3x10GE SFP + 3x10GE SFP + 3x10GE SFP +
FIA
3x 10G
3x 10G
Typhoon
FIA
3x 10G
Typhoon
FIA
3x 10G
Typhoon
Switch Fabric ASIC
3x10GE
CPU
3x 10G
8x55 G
RSP 3 Switch Fabric
Switch Fabric RSP0
3x 10G
Switch Fabric
Typhoon
FIA
3x 10G
Typhoon
Original packet format © 2013 Cisco and/or its affiliates. All rights reserved.
RSP1
Super-frame format (unicast only) between switch fabric and FIA, fabric and fabric 22
CPU
6x 10G
6x10GE Hex PHY
Typhoon
FIA
6x 10G
Typhoon
RSP 3 Switch Fabric
FIA
6x 10G
6x10GE Hex PHY
Typhoon
FIA
6x 10G
6x10GE Hex PHY
Typhoon
FIA
Typhoon
FIA
6x 10G
6x10GE Hex PHY 6x 10G
6x10GE Hex PHY
Typhoon
FIA
Switch Fabric ASIC
6x10GE Hex PHY
8x55 G
RSP 3 Switch Fabric
Switch Fabric RSP0
Switch Fabric RSP1
36x10G line card © 2013 Cisco and/or its affiliates. All rights reserved.
23
CPU 100G
FIA
Egress Typhoon
FIA
100GE MAC/PHY
RSP 3 Switch Fabric 100G
100G
Ingress Typhoon
FIA
100GE MAC/PHY 100G
Switch Fabric ASIC
Ingress Typhoon
8x55 G
RSP 3 Switch Fabric
Switch Fabric RSP0
Switch Fabric
Egress Typhoon
FIA
RSP1
MUX FPGA © 2013 Cisco and/or its affiliates. All rights reserved.
24
CPU Supported MPA
FIA
Typhoon
FIA
1x40GE 2x40GE RSP 3 Switch Fabric 2x10GE
4x10GE 20xGE Supported MPA
Typhoon
FIA
8x55 G Switch Fabric ASIC
Typhoon
RSP 3 Switch Fabric
Switch Fabric RSP0
1x40GE 2x40GE 2x10GE 4x10GE
Typhoon
Switch Fabric
FIA
RSP1
20xGE
Modular line card © 2013 Cisco and/or its affiliates. All rights reserved.
25
CPU Supported MPA
RSP 3 Switch Fabric 2x10GE
Typhoon
FIA
4x10GE 20xGE Supported MPA 1x40GE
Typhoon
Switch Fabric ASIC
1x40GE
8x55 G
RSP 3 Switch Fabric
Switch Fabric RSP0
FIA Switch Fabric
2x10GE 4x10GE
RSP1
20xGE
Modular line card © 2013 Cisco and/or its affiliates. All rights reserved.
26
Same as existing system: Two-stage IOS-XR packet forwarding Uniform packet flow: All packet go through central fabric on the RP
3x 10G 3x10GE SFP + 3x 10G 3x10GE SFP + 3x 10G
3x 10G 3x10GE SFP + 3x 10G 3x10GE SFP + 3x 10G 3x10GE SFP + 3x 10G 3x10GE SFP +
FIA FIA
Typhoo n Typhoo n Typhoo n Typhoo n
FIA
FIA
Switch Fabric
Typhoo n Typhoo n
© 2013 Cisco and/or its affiliates. All rights reserved.
FIA
Switch Fabric
Switch Fabric ASIC
3x 10G 3x10GE SFP +
Typhoo n
Switch Fabric ASIC
3x10GE SFP +
Typhoo n
Ingress 100 Typhoon G
FIA
Egress 100 Typhoon G
FIA
Ingress Typhoon 100
100GE MAC/ PHY
G
100GE MAC/ PHY FIA
Egress Typhoon 100 G
27
ASR 9001 System Architecture Overview MPAs 2,4x10GE 20xGE 1x40GE
SFP+ 10GE On-board 4x10 SFP+ ports
SFP+ 10GE
Interna l EOBC
SFP+ 10GE
LC CPU
SFP+ 10GE MPAs 2,4x10GE 20xGE 1x40GE
Typhoon
RP CPU
Switch Fabric ASIC
FIA
Typhoon
FIA
It has both central RP and LC CPU like big chassis But it only have central switch fabric, no LC fabric Maximum 120Gbps bi-directional system
© 2013 Cisco and/or its affiliates. All rights reserved.
28
ASR 9001 Packet Flow Overview Supported MPA
Same as big chassis system: Two-stage IOS-XR packet forwarding
2,4x10GE 20xGE 1x40GE SFP+ 10GE SFP+ 10GE
Interna l EOBC
SFP+ 10GE
LC CPU
SFP+ 10GE Supported MPA
Typhoon
RP CPU
Switch Fabric ASIC
FIA
Typhoon
FIA
2,4x10GE 20xGE 1x40GE © 2013 Cisco and/or its affiliates. All rights reserved.
29
§ 3 memory options for each line card: § Extended (or high queue) § Base (medium queue) § Low (low queue)* § Different memory options have different: § QoS queue scale § L2 sub-interface scale § All other system wide scale parameters remain same: § FIB § MAC address § Bridge-domain § L3 sub-interface § VRF, etc § All line cards have the same HW à Identical features § Mixed LC types supported on same chassis
High Medium Low
* A9K-16T/8 only have “B” option. It doesn’t have “E” or “L” option as of the 3.9.1 release © 2013 Cisco and/or its affiliates. All rights reserved.
30
Shared between L2/L3 on Trident. Dedicated mem on Typhoon
NP complex FIB
STATS MEMORY
MAC
LOOKUP MEMORY
Network Process Unit
-
FRAME MEMORY TCAM
§ Each NPU has Four Main memories: – Lookup/Search Memory (RLDRAM): stores MAC, FIB, and Adjacencies Tables – TCAM: classification (Vlan Tag (EVCs), QoS and Security ACL – Stats QDR memory: interface and forwarding statistics, policers data, etc – Frame memory: buffer memory for Queues § 3 LC versions – low, base and extended - differ for size of memories – TCAM, QDR and Frame memory sizes depend on LC version Affects number of QoS queues and L2 sub-interfaces supported – Search Memory is same System level scale (unicast, multicast, MPLS label) adjacency and MAC address) not affected by a mix of LCs © 2013 Cisco and/or its affiliates. All rights reserved.
31
Low Queue
Medium Queue
High Queue
512K
512K
512K
1M/128K
1M/128K
1M/128K
32K
32K
32K
512K/128K
512K/128K
512K/128K
VRFs
4k
4k
4k
L3 Subif/Port
4k
4k
4k
Bridge Domains
8k
8k
8k
128k
128k
128k
4k
16k
32k
Queues
8/port
64k/32k
256k/128k
Policers
8k
128k
256k
MAC Addresses IPv4 Routes (total/per VRF) ARP entries IPv6 Routes (total/per VRF)
MPLS Labels EFPs (L2 sub-int)
© 2013 Cisco and/or its affiliates. All rights reserved.
Different
Metric
Common
Capability Comparison
32
Feature
-TR
-SE
Comments
FIB (V4+V6)
4M
Multicast FIB
128K
MAC
2M
Support per-LC MAC learning in the future
L3 VRF
4K
8K in 4.2.1
BD/VFI
64K
PW
128K 8K/LC
20K/LC
L2 interface
16K/LC
64K/LC
QoS
8 queues/port (I and O) 8K policers/NP 1G frame memory/NP
256K queues (I+O) / NP 256K policers/NP 2G frame memory/ NP
ACL
24k ACE
96k ACE
© 2013 Cisco and/or its affiliates. All rights reserved.
Per-LC scale
L3 interface
System wide scale
V4 and V6 share the same table V6 uses two FIB entries Support per-VRF FIB table download per LC (SVD)
Note XR431 has ACL compression (hybrid) 33
How are you using the ASR9000? a) Mainly in an L2 PE environment b) As a replacement for a 7600 c) Mainly as a L3 PE d) A core router e) Anything it can do
© 2013 Cisco and/or its affiliates. All rights reserved.
34
§ Centralized fabric resides on RSP § Logically separated from LC and RSP § All fabric ASICs run in active mode regardless of RSP Redundancy status § Extra fabric bandwidth and instant fabric switch over § Each 40G LC/RSP has one fabric interface ASIC § 80G line rate LCs have 2 fabric interface ASICs Crossbar Fabric ASIC Crossbar Fabric ASIC
23Gbps per fabric channel
Dual RSP: 4x23Gbps =184Gbps Single RSP: 4x23Gbps=92Gbps
Arbitration Fabric Interface and VOQ
Single-Fabric interfaces 40G Linecard
Dual RSP: 4x23Gbps =92Gbps Single RSP: 2x23Gbps=46Gbps © 2013 Cisco and/or its affiliates. All rights reserved.
RSP0 Crossbar Fabric ASIC Crossbar Fabric ASIC
Fabric Interface and VOQ
Fabric Interface and VOQ
Dual-Fabric interfaces 80G Linecard
Arbitration
RSP1
35
“0” packet loss guarantee during RSP failover and OIR § Access to fabric controlled using central arbitration. § One Arbitration ASIC (Arbiter) per RSP § Both Arbiters work in parallel – both answer to requests to transmit § FIAs follow active Arbiter, and switch to backup if needed § Arbiter switchover controlled by low level hardware signalling
Arbitration - Relative to a egress NPU - QoS aware
Crossbar Fabric ASIC
Fabric is fully non blocking
Crossbar Fabric ASIC Arbitration
Fabric Interface and VOQ
Single-Fabric interfaces 40G Linecard
RSP0 Crossbar Fabric ASIC Crossbar Fabric ASIC
Fabric Interface and VOQ
Fabric Interface and VOQ
Dual-Fabric interfaces 80G Linecard
Arbitration
© 2013 Cisco and/or its affiliates. All rights reserved.
RSP1
36
RSP0
1: Fabric Request
Crossbar Fabric ASIC
5: credit return
Crossbar Fabric ASIC Arbitration Fabric Interface and VOQ
2: Arbitration
3: Fabric Grant 4: load-balanced transmission across fabric links
Fabric Interface and VOQ
Crossbar Fabric ASIC Crossbar Fabric ASIC Arbitration
RSP1
© 2013 Cisco and/or its affiliates. All rights reserved.
37
Crossbar Fabric ASIC Crossbar Fabric ASIC Arbitration Fabric Interface and VOQ
4
RSP0
Fabric Interface and VOQ
4
3
2
1
Crossbar Fabric ASIC Crossbar Fabric ASIC Arbitration
RSP1
§
Unicast traffic sent across first available fabric link to destination (maximizes efficiency)
§
Each frame (or superframe) contains sequencing information
§
All destination fabric interface ASIC have re-sequencing logic
§
Additional re-sequencing latency is measured in nanoseconds
© 2013 Cisco and/or its affiliates. All rights reserved.
38
Crossbar Fabric ASIC Crossbar Fabric ASIC Arbitration Fabric Interface and VOQ
RSP0
Fabric Interface and VOQ
C 1
B 2
A 3
B 1
A 2
A 1
Flows exit in-order Crossbar Fabric ASIC Crossbar Fabric ASIC Arbitration
RSP1
§ § §
Multicast traffic hashed based on (S,G) info to maintain flow integrity Very large set of multicast destinations preclude re-sequencing Multicast traffic is non arbitrated – sent across a different fabric plane
© 2013 Cisco and/or its affiliates. All rights reserved.
39
• Multiple unicast frames from/to same destinations aggregated into one
super frame
• Super frame is created if there are frames waiting in the queue, up to 32
frames or when min threshold met, can be aggregated into one super frame
• Super frame only apply to unicast, not multicast • Super-framing significantly improves total fabric throughput
Packet 2
Packet 3
Packet 2
Packet 1
Max Super-frame
© 2013 Cisco and/or its affiliates. All rights reserved.
Min Super-frame
Packet 1
No super-framing
Packet 1
Min reached
Packet 1
Max reached Jumbo
0 (Empty)
40
End-to-End priority (P1,P2, Best-effort) propagation à Guarantee bandwidth, low latency for high priority traffic at any congestion point 3 strict priority level across all internal HW components Ingress side of LC
PHY
NP0
PHY
NP1
PHY
NP2
PHY
NP3
Egress side of LC
CPU
Configure with Ingress MQC 4-layer hierarchy Two strict high priority + Normal priority © 2013 Cisco and/or its affiliates. All rights reserved.
FIA
FIA Switch Fabric
2
Ingress (sub-)interface QoS Queues
NP0
3
2
1
1
CPU
3
Virtual Output Queues
Egress FIA Queues
Implicit Configuration Two strict high priority + Normal priority
4
PHY
NP1
PHY
NP2
PHY
NP3
PHY
4
Egress (sub-)interface QoS Queues Configure with Egress MQC 4-layer hierarchy Two strict high priority + Normal priority 41
VoQ Scale: Each FIA has P1/P2/BE queue set for every NP and RSPs in the entire system
Egress NP congestion à à backpressure to ingress FIA à Packet is en-queued in the dedicated VoQ à No impact of the packet going to different egress NP à No head-of-line-block issue Backpressure: egress NP à egress FIA à fabric Arbiter à ingress FIA à VoQ
Ingress side of LC1
PHY
10Gbps 5Gbps
PHY
5Gbps
NP0
Egress side of LC2
CPU
CPU
NP1
PHY
NP2
PHY
NP3
FIA
FIA Switch Fabric
1
NP0
PHY
NP1
PHY
NP2
PHY
NP3
PHY
2
3
Packet going to different egress NP put into different VoQ set à Congestion on one NP won’t block the packet going to different NP © 2013 Cisco and/or its affiliates. All rights reserved.
42
From wire
Ingress linecard
I/F classificatio n
*IFIB action
Security ACL classification
QoS action
QOS classification
Fwd lookup
*IFIB lookup
L2 rewrite
QOS policer action
Security ACL action
To fabric
From fabric egress linecard
Security ACL action QoS+ policer action © 2013 Cisco and/or its affiliates. All rights reserved.
QOS classification
Security ACL classification
L2 rewrite
Fwd lookup
To wire 43
From wire
Ingress linecard Ingress
I/F classificatio n
Security ACL classification
*IFIB action
QoS action
To fabric
ACL action
QOS classification
© 2013 Cisco and/or its affiliates. All rights reserved.
Fwd lookup
Queue/ shape/ Mark QOS WRED policer L2 rewrite action
*IFIB lookup Security ACL action
From fabric
Egress QoS Action
egress linecard WRED classifies on ACLmarked/remarked values L2 rewrite QOS classification Queue/ classification
Police
QoS action
QoS Action
Mark
Fwd lookup
shape/ WRED
To wire 44
5 Stages:
Parse
Search
Resolve
Modify
Queueing Scheduling
• L2/L3 header packet parsing in TCAM • Builds keys for ingress ACL, QoS and forwarding lookups (uCode)
• Performs QoS and ACL lookups in TCAM tables • Performs L2 and L3 lookups in RLDRAM
• Processes Search results: • ACL filtering • Ingress QoS classification and policing • Forwarding (egress SFP determined) • Performs L2 MAC learning
• Adds internal system headers • Egress Control Header (ECH) • Switch Fabric Header (SFH)
• Queuing, Shaping and Scheduling functions
© 2013 Cisco and/or its affiliates. All rights reserved.
45
5 Stages:
Parse
Search
Resolve
Modify
Queueing Scheduling
• L2/L3 header packet parsing in TCAM • Builds keys for egress ACL, QoS and forwarding lookups (uCode)
• Performs QoS and ACL lookups in TCAM tables • Performs L2 and L3 lookups in RLDRAM
• Processes Search results: • ACL filtering • Egress QoS classification and policing • Forwarding (egress Port determined) • Performs L2 MAC learning
• L2 hdr rewrite • L2/L3 QoS packet settings
• Queuing, Shaping and Scheduling functions
© 2013 Cisco and/or its affiliates. All rights reserved.
46
Packet Flow (Simplified) from wire LAGID
lookup key L3: (VRF-‐ID, IP DA) TCAM
rxIDB
L3FIB
rx-adj
Packet classifica,on
Source interface info
L3 FIB lookup
Next-‐hop
Ingress NPU
Rx LAG hashing LAG
SFP
Packet rewrite System headers added rewrite
SFP
ECH Type: L3_UNICAST
Switch Fabric Port (egress NPU) SFP
ACL and QoS Lookup also happen in parallel Fabric
Tx LAG hashing LAG rewrite
Egress NPU to wire © 2013 Cisco and/or its affiliates. All rights reserved.
txIDB
tx-adj
L3FIB
des,na,on interface info
Next-‐hop
L3 FIB lookup
ECH Type: L3_UNICAST
=> L3FIB lookup
ACL and QoS Lookup happens before rewrite ECH type: tell egress NPU type of lookup it should execute 47
Getting SFP for a port
Shows uIDB information
RP/0/RSP1/CPU0:asr#sh controllers pm interface gig 0/0/0/1 loc 0/0/CPU0 Tue Aug 3 13:20:19.853 UTC Ifname(1): GigabitEthernet0_0_0_1, ifh: 0x40000c0 : iftype 0xf egress_uidb_index 0x3 ingress_uidb_index 0x3 Ports connected to the port_num 0x1 same NPU share the phy_port_num 0x1 same SFP value channel_id 0x3 lag_id 0x0 virtual_port_id 0x0 switch_fabric_port 0x3 in_tm_qid_fid0 0x10000202 in_tm_qid_fid1 0xffffffff in_qos_drop_base 0x10600 out_tm_qid_fid0 0x82 out_tm_qid_fid1 0xffffffff out_qos_drop_base 0x38480 bandwidth 1000000 kbps ing_stats_ptrs 0x0, 0x0, 0x0, 0xd8414 egr_stats_ptrs 0x0, 0x0, 0x0, 0xd8423 l2_transport 0x0 ac_count 0x0 parent_ifh 0x0 parent_bundle_ifh 0x0 © 2013 Cisco and/or its affiliates. All rights reserved.
48
L3FIB
lookup key L3: (VRF-‐ID, IP DA)
RDT: Reduced direct tree
VRF 0
0 1
VRF
Recursive Prefix Leaf
Recursive LDI
1
2
VRF, IP DA 8MSB
…
Direct Lookup
2 … …
4
25 6
15 ... …
IP DA 24LSB
Non-Recursive Prefix Leaf
Adjacency Non-Recursive pointer LDI
Tree Lookup
IP DA (32bits) Adjacency Non-Recursive pointer LDI
Tree Lookup
… 4k
Recursive Prefix Leaf © 2013 Cisco and/or its affiliates. All rights reserved.
Recursive LDI 49
Leaf: 1 per IPv4 prefix Endpoint of IPv4 lookup Points to LDI
If VRF < 15, 2 steps route lookup for faster search: • direct lookup key look up based on 8 MSBs of IP DA RDT: Reduced direct tree • Tree Ilookup L3: (VRF-‐ID, P DA) based on remaining 24 LSBs
VRF 0
0 1
VRF
IP DA 24LSB
2 …
Direct Lookup
NR NR NR … LDI LDI LDI
….
2 … …
4
25 6
15
4ways Tree Lookup
...
Recursive Prefix Leaf
R R R R LDI LDI LDI LDI
IP DA (32bits) Tree Lookup
… 4k
VRF based array lookup
Adjacency pointers 32ways (4.0.1) (1xLDI)
1 VRF, IP DA 8MSB
…
Non-Recursive Prefix Leaf
L3FIB
If VRF > 15: • Tree lookup based on 32 bits IP address
© 2013 Cisco and/or its affiliates. All rights reserved.
Adjacency Non-Recursive pointer LDI: Loadinfo LDI Recursive or nonrecursive Nonrecursive created per-IGP prefix • up to 8 per prefix (ECMP) – 32 (1HCY11) Recursive (BGP) can be shared by Recursive Recursive leaves Prefixmultiple Leaf prefixLDI 50
L3FIB
32ways (4.0.1) Non-Recursive Prefix Leaf
Adjacency pointer(s) 1xLDI
NR NR NR … LDI LDI LDI
OIF
Adj
Adj
LAG
OIF
Protected TE Adj
LAG
OIF
8ways Recursive Prefix Leaf
R R R R LDI LDI LDI LDI Adjacency Non-Recursive pointer LDI
Backup TE Adj Recursive Prefix Leaf © 2013 Cisco and/or its affiliates. All rights reserved.
OIF
Recursive LDI 51
L3FIB
32ways (4.0.1) Non-Recursive Prefix Leaf
Adjacency pointer(s) 1xLDI
NR NR NR … LDI LDI LDI
If outgoing interface goes down (with ECMP), only NR-LDI has to change. IGP/BGP routes (prefixes) are untouched 4ways (4.0.1) Recursive Prefix Leaf
R R R R LDI LDI LDI LDI
Adj OIF If bundle member port goes down, only LAG table has to change
Adj
LAG
OIF
Protected TE Adj
LAG
OIF
Adjacency Non-Recursive pointer LDI For TE-FRR switchover, only protected TE adjacency changes Recursive Prefix Leaf © 2013 Cisco and/or its affiliates. All rights reserved.
Backup TE Adj
OIF
Recursive LDI 52
1: IPv4 Unicast or IPv4 to MPLS – No or unknown Layer 4 protocol: IP SA, DA and Router ID
IPv6 uses first 64 bits in 4.0 releases, full 128 in 42 releases
– UDP or TCP: IP SA, DA, Src Port, Dst Port and Router ID 2: IPv4 Multicast – For (S,G): Source IP, Group IP, next-hop of RPF – For (*,G): RP address, Group IP address, next-hop of RPF 3: MPLS to MPLS or MPLS to IPv4 – # of labels 4 : 4th label and Router ID
-
L3 bundle uses 5 tuple as “1” (eg IP enabled routed bundle interface)
-
MPLS enabled bundle follows “3”
-
L2 access bundle uses access S/D-MAC + RID, OR L3 if configured (under l2vpn)
-
L2 access AC to PW over mpls enabled core facing bundle uses PW label (not FAT-PW label even if configured) -
FAT PW label only useful for P/core routers
© 2013 Cisco and/or its affiliates. All rights reserved.
53
LDP
RSVP-TE
Static
LSD
BGP
OSPF
ISIS
EIGRP
RIB
RSP CPU
ARP SW FIB
AIB
Adjacency LC NPU
LC CPU
© 2013 Cisco and/or its affiliates. All rights reserved.
FIB
AIB: Adjacency Information Base RIB: Routing Information Base FIB: Forwarding Information Base LSD: Label Switch Database
54
Show commands
LDP
RSVP-TE
Static
LSD
BGP
OSPF
ISIS
EIGRP
RIB
RSP CPU
RP/0/RSP0/CPU0:asr#sh route 222.0.0.6/31 Routing entry for 222.0.0.6/31 Known via "isis isis1", distance 115, metric 20, type level-1 ARP Installed Mar 2 17:58:12.251 for 00:00:47 FIB Adjacency FIB Routing Descriptor SW Blocks 222.0.0.2, from 222.2.2.1, via TenGigE0/1/0/3 AIB Route metric is 20 LC NPU No advertising protos. AIB: Adjacency Information Base LC CPU RIB: Routing Information Base FIB: Forwarding Information Base © 2013 Cisco and/or its affiliates. All rights reserved. LSD: Label Switch Database
55
Show commands RP/0/RSP0/CPU0:asr#show adjacency summary location 0/1/CPU0 BGP OSPF LDP RSVP-TE Adjacency table (version 26) has 19Static adjacencies: ISIS EIGRP 11 complete adjacencies 8 incomplete adjacencies 0 deleted adjacencies in quarantine list 8 adjacencies LSDof type IPv4 RIB RSP CPU 8 complete adjacencies of type IPv4 0 incomplete adjacencies of type IPv4 0 deleted adjacencies of type IPv4 in quarantine list 0 interface adjacencies of type IPv4 4 multicast adjacencies of type IPv4 ARP SW FIB
AIB
Adjacency LC NPU
LC CPU
© 2013 Cisco and/or its affiliates. All rights reserved.
FIB
AIB: Adjacency Information Base RIB: Routing Information Base FIB: Forwarding Information Base LSD: Label Switch Database
56
Show commands
BGP OSPF LDP RSVP-TEcef 222.0.0.6 location 0/1/CPU0 RP/0/RSP0/CPU0:viking-1#sh Static 222.0.0.6/31, version 1, internal 0x40000001 ISIS EIGRP Updated Mar 2 17:58:11.987 local adjacency 222.0.0.2 Prefix Len 31, traffic index 0, precedence routine (0) RIB weight 0, via 222.0.0.2, LSD TenGigE0/1/0/3, 5 dependencies, class 0 next hop 222.0.0.2 local adjacency
RSP CPU
ARP SW FIB
AIB
Adjacency LC NPU
LC CPU
© 2013 Cisco and/or its affiliates. All rights reserved.
FIB
AIB: Adjacency Information Base RIB: Routing Information Base FIB: Forwarding Information Base LSD: Label Switch Database
57
Show commands RP/0/RSP0/CPU0:asr#sh cef 222.0.0.6 hardware ingress lo 0/1/CPU0 222.0.0.6/31, version 1, internal 0x40000001 (0xb1d66c6c) [1], 0x0 (0xb1b4f758), 0x0 (0x0) Updated Mar 2 17:58:11.987 local adjacency 222.0.0.2 BGP OSPF RSVP-TE Prefix Len 31, LDP traffic index 0, precedence routine (0) Static via 222.0.0.2, TenGigE0/1/0/3, 5 dependencies, weight 0, class 0 ISIS EIGRP next hop 222.0.0.2 local adjacency EZ:0 Leaf ============ Search ctrl-byte0: Leaf Action :
LSD 0x3
RIB ctrl-byte1:
0x8
RSP CPU
ctrl-byte2:0x5
FORWARD
prefix 31: Search length Control:Flags match : done : ARP ext_lsp_array : recursive : default_action:
1 0 0 0 1
valid: 1 ifib_lookup: 0 match_all_bit: 0 nonrecursive : 1
SW FIB
AIB
Non Recursive Leaf: -------------------
LC ldi ptr : 10936 (0x2ab8) igpCPU statsptr:0 rpf ptr : 0x0000 BGP policy a/c : 0 AS number : 0 © 2013 Cisco and/or its affiliates. All rights reserved.
FIB
Adjacency LC NPU
AIB: Adjacency Information Base RIB: Routing Information Base FIB: Forwarding Information Base LSD: Label Switch Database
58
•
Cisco Support Community XR OR and Platforms https://supportforums.cisco.com/community/netpro/service-providers/ios-xr_
•
ASR9000/XR Feature Order of operation
•
ASR9000/XR Frequency Synchronization
•
ASR9000/XR: Understanding SNMP and troubleshooting
•
Cisco BGP Dynamic Route Leaking feature Interaction with Juniper
•
ASR9000/XR: Cluster nV-Edge guide
•
Using COA, Change of Authorization for Access and BNG platforms
•
ASR9000/XR: Local Packet Transport Services (LPTS) CoPP
•
ASR9000/XR: How to capture dropped or lost packets
•
ASR9000/XR Understanding Turboboot and initial System bring up
•
ASR9000/XR: The concept of a SMU and managing them
•
ASR9000/XR Using MST-AG (MST Access Gateway), MST and VPLS
•
ASR9000/XR: Loadbalancing architecture and characteristics
•
ASR9000/XR Netflow Architecture and overview
•
ASR9000 Understanding the BNG configuration (a walkthrough)
•
ASR9000/XR NP counters explained for up to XR4.2.1
•
ASR9000/XR Understanding Route scale
•
ASR9000/XR Understanding DHCP relay and forwarding broadcasts
•
ASR9000/XR: BNG deployment guide
59
59
•
ASR9000/XR: Understanding and using RPL (Route Policy Language)
•
ASR9000/XR What is the difference between the -p- and -px- files ?
•
ASR9000/XR: Migrating from IOS to IOS-XR a starting guide
•
ASR9000 Monitoring Power Supply Information via SNMP
•
ASR9000 BNG Training guide setting up PPPoE and IPoE sessions
•
ASR9000 BNG debugging PPPoE sessions
•
ASR9000/XR : Drops for unrecognized upper-level protocol error
•
ASR9000/XR : Understanding ethernet filter strict
•
ASR9000/XR Flexible VLAN matching, EVC, VLAN-Tag rewriting, IRB/BVI and defining L2 services
•
ASR9000/XR: How to use Port Spanning or Port Mirroring
•
ASR9000/XR Using Task groups and understanding Priv levels and authorization
•
ASR9000/XR: How to reset a lost password (password recovery on IOS-XR)
•
ASR9000/XR: How is CDP handled in L2 and L3 scenarios
•
ASR9000/XR : Understanding SSRP Session State Redundancy Protocol for IC-SSO
•
ASR9000/XR: Understanding MTU calculations
•
ASR9000/XR: Troubleshooting packet drops and understanding NP drop counters
•
Using Embedded Event Manager (EEM) in IOS-XR for the ASR9000 to simulate ECMP "min-links"
•
XR: ASR9000 MST interop with IOS/7600: VLAN pruning
60
60
Use the Q&A panel to submit your questions. Experts will start responding those
© 2013 Cisco and/or its affiliates. All rights reserved.
61
Expert responding some of your questions verbally. Use the Q&A panel to continue asking your questions
© 2013 Cisco and/or its affiliates. All rights reserved.
62
Those who fill out the Evaluation Survey will enter a raffle to win:
$50 Amazon Gift Card
To complete the evaluation, please click on link provided in the chat or in the pop-up once the event is closed.
© 2013 Cisco and/or its affiliates. All rights reserved.
63
What do Cisco Router Technology, McDonalds and Disney have in common? In this same year Cisco earned its first patent for its method and apparatus for routing communications among computer networks. The first McDonalds in Beijing China opened and Euro Disney opened in France.
What year was it? a) 1992 b) 1994 c) 1995
64
If you have additional questions, you can ask them to Xander. He will be answering from May 14 to May 24, 2013 https://supportforums.cisco.com/thread/2216914 You can watch the video or read the Q&A 5 business days after the event at https://supportforums.cisco.com/community/netpro/ask-the-expert/ webcasts
© 2013 Cisco and/or its affiliates. All rights reserved.
65
in Russian Configuration and Features of Border Gateway Protocol (BGP) Prefix-Independent Convergence Tuesday May 21st at 10:00 a.m. Brussels 12:00 p.m. Moscow With Cisco expert: Nataliya Omelyanyuk
During this live event session explains the advantages and features of Border Gateway Protocol (BGP) prefix-independent convergence technology in service provider networks. She will also provide configuration examples for Cisco IOS Software and Cisco IOS XR Software.
Join the discussion for these Ask The Expert webcasts at: https://supportforums.cisco.com/community/netpro/expert-corner#view=webcasts © 2013 Cisco and/or its affiliates. All rights reserved.
66
in Japanese Cisco Catalyst High CPU Troubleshooting Training Tuesday May 28th 10:00 a.m. Japan Standard Time
Tuesday May 27th 6:00 p.m. PDT (San Francisco)
With the Cisco expert: Yasuhiro Nakajima
This session discusses several troubleshooting examples for High CPU Utilization on Cisco Catalyst Series. Nakajima will note major checkpoints and specific ways to narrow down the problems on this issue. Join the discussion for these Ask The Expert webcasts at: https://supportforums.cisco.com/community/netpro/expert-corner#view=webcasts
© 2013 Cisco and/or its affiliates. All rights reserved.
67
in English Configuration, Design, and Troubleshooting of Cisco Nexus 1000 Tuesday, June 4 at 7:00 a.m. PDT (San Francisco) 10:00 a.m. EDT (New York) 3:00 p.m. Paris
With Cisco expert: Louis Watta
During this live event, Cisco expert Louis Watta will go over the design, configuration, and troubleshooting of Cisco Nexus 1000V Series Switches operating inside VMware ESXi and Hyper-V..
Join the discussion for these Ask The Expert webcasts at: http://tools.cisco.com/gems/cust/customerQA.do? METHOD=E&LANGUAGE_ID=E&SEMINAR_CODE=S18361&PRIORITY_CODE= © 2013 Cisco and/or its affiliates. All rights reserved.
68
English Topic: Fiber Channel over Ethernet (FCoE) With Cisco expert Ozden Karakok Learn how to design, plan, configure, implement, and troubleshoot Fibre Channel over Ethernet Ends May 17, 2013 Topic: SSO with CWMS, IRP and ELM Solutions With Cisco expert Arun Kumar Learn and ask questions about Cisco WebEx Meetings Server (CWMS) Ends May 17, 2013 Topic: Understanding, configuring and troubleshooting IP Multicast and MVPN With Cisco expert Pulikkal Sekharan Raju Learn and ask questions on how to deploy, configure and troubleshoot Single Sign On (SSO), Internet Reverse Proxy (IRP), Enterprise License Manager (ELM) Starts May 20, 2013 Topic: Deploying Cisco FabricPath in Data Center NetworkFabricPath With Cisco Anees Mohamed/Viral Bhutta Learn and ask questions about how to plan, design, and implement Cisco Overlay Transport Virtualization (OTV) in your Data Center Network. Starts May 20, 2013
Join the discussion for these Ask The Expert Events at: https://supportforums.cisco.com/community/netpro/expert-corner#view=ask-the-experts © 2013 Cisco and/or its affiliates. All rights reserved.
69
June 25-27 Cisco Live 2013 Orlando – Virtual Event Full agenda including live WebEx sessions will be published at the end of May.
Agenda & Details: https://www.ciscolive365.com/connect/agenda.ww
© 2013 Cisco and/or its affiliates. All rights reserved.
70
https://supportforums.cisco.com http://www.facebook.com/CiscoSupportCommunity http://twitter.com/#!/cisco_support http://www.youtube.com/user/ciscosupportchannel https://plus.google.com/110418616513822966153? prsrc=3#110418616513822966153/posts http://itunes.apple.com/us/app/cisco-technical-support/id398104252?mt=8 https://play.google.com/store/apps/details?id=com.cisco.swtg_android http://www.linkedin.com/groups/CSC-Cisco-Support-Community-3210019 Newsletter Subscription: https://tools.cisco.com/gdrp/coiga/showsurvey.do? surveyCode=589&keyCode=146298_2&PHYSICAL%20FULFILLMENT%20Y/ N=NO&SUBSCRIPTION%20CENTER=YES © 2013 Cisco and/or its affiliates. All rights reserved.
71
If you speak Spanish, Portuguese, Japanese, Polish or Russian, we invite you to ask your questions and collaborate in your language: • Spanish à https://supportforums.cisco.com/community/spanish • Portuguese à https://supportforums.cisco.com/community/portuguese • Japanese à https://supportforums.cisco.com/community/csc-japan • Polish à https://supportforums.cisco.com/community/etc/netpro-polska • Russian à https://supportforums.cisco.com/community/russian
© 2013 Cisco and/or its affiliates. All rights reserved.
72
What do Cisco Router Technology, McDonalds and Disney have in common? In this same year Cisco earned its first patent for its method and apparatus for routing communications among computer networks. The first McDonalds in Beijing China opened and Euro Disney opened in France.
What year was it? a) 1992 b) 1994 c) 1995
73
Thank You for Your Time Please Take a Moment to Complete the Evaluation
Thank you.
