ASR9000 Troubleshooting Architectures Xander Thuijs CCIE#6775 Principal Engineer BRKSPG-2904

Agenda •

Introduction •

What is new and what has changed since we last spoke

•

Understanding TCAM operations and ACLs

•

More on Troubleshooting Packet Forwarding •

Software forwarding/handling • Capturing packets with SPP

•

Fragmentation Identifying packet drops in the system Loadbalancing reviewed

• •

•

Understanding QOS aspects • •

•

Tomahawk, what’s changed? • • •

•

Queue limits and buffering Monitoring

Understanding the HW Show commands CPAK

More on Usability, XR strategy, SMU’s and system management

3

Acknowledgements •

With contributions from •

Aleks Vidakovic; Tomahawk updates • Sadanande Phadke; TCAM updates • Eddie Chami; usability and software manager •

YOU Our boys really REALLY appreciate the interaction online via the forums and the direct feedback that you have given. We hope to have shown you the last few years that your feedback didn’t get unnoticed with a large emphasis on usability and ease of use. Your ratings on this session the last few years has been overwhelming which gives us so much pride and joy to continue and improve whatever we need to or can. So PLEASE keep that input coming and YOUR participation is necessary to build a better product TOGETHER! 4

Introduction What is new and what has changed recently

5

New hardware •

ASR9910 •

•

Beefed up backplane for higher speed cards (8x100)

Wildchild

SFP SFP SFP SFP SFP SFP

•

Fixed 40G/56G linecard • 40x1G or 2x10G+16x1G

SFP

P H Y

CPU

SFP

SFP SFP SFP SFP SFP SFP SFP

P H Y

Switch Fabric

SFP

•

VSM applications

SFP SFP SFP SFP SFP SFP

•

Radware/Arbor/IPSEC

SFP SFP

SFP SFP SFP SFP

•

SFP

Tomahawk linecards

SFP SFP

4x100G/8x100G • RSP880

P H Y

RSP0

NP

FIA

Switch Fabric

SFP

SFP

•

P H Y

SFP SFP SFP SFP SFP SFP SFP

P H Y

Switch Fabric RSP1

6

Organizational and Software •

No more “BU”. Routing segment taking care of all routing platforms •

The goods from every group and architecture put together, eliminating the bad from before

•

Dedicated dev/test teams focusing on usability

•

Extreme focus on usability, ease of use and user experience •

Installation • Knobs, tweaks and tunes •

ASR 9000 Product Survey ‘What can be improved?’

Feature packs / Service packs •

Improving documentation (hopefully? )

•

More forum stuff • different model for cco documentation 7

Troubleshooting Packet Forwarding Update and recap

8

NPU Packet Processing - Ingress 5 Stages:

Parse •L2/L3 header packet parsing in TCAM •Builds keys for ingress ACL, QoS and forwarding lookups (uCode)

Search

Resolve

Modify

•Performs QoS and ACL lookups in TCAM tables •Performs L2 and L3 lookups in RLDRAM

•Processes Search results: •ACL filtering •Ingress QoS classification and policing •Forwarding (egress SFP determined) • Performs L2 MAC learning

•Adds internal system headers •Egress Control Header (ECH) •Switch Fabric Header (SFH)

Queueing Scheduling

•Queuing, Shaping and Scheduling functions •All packets go through this stage

9

ASR9000 Fully Distributed Control Plane LPTS (local packet transport service): control plane policing

CPU

Punt FPGA

RP FIA

CPU

iFIB NP LPTS

FIA

Control packet

PHY

NP

FIA

CPAK 1 CPAK 2 CPAK 3 CPAK 4

PHY

NP

CPAK 5

Switch Fabric (SM15)

Switch Fabric

Up to 14x120G

CPAK 6

PHY CPAK 7

FIA

LC1

…

PHY

CPAK 0

NP

FIA

Switch Fabric

RP CPU: Routing, MPLS, IGMP, PIM, HSRP/VRRP, etc LC CPU: ARP, ICMP, BFD, NetFlow, OAM, etc

10

Troubleshooting NP Forwarding 1.

Identify interface in question with problem.

2.

Identify the mapping from interface to NPU.

3.

Examine NP counters.

4.

Look for rate counters that match lost traffic rate. •

If none of the counters match the expect traffic, check drops at interface controller

5.

Lookup the counter description.

6.

If required capture the packet hitting the counter (Typhoon only).

7.

If packets are forwarded to the fabric, run fabric troubleshooting steps.

8.

Identify egress NP and repeat steps 3 to 6. 11

“Show drops all” enhancement •

Supported starting with 5.3.0

•

Uses a ‘grammar’ file to combine outputs of other show commands •

•

Easy way to achieve a combined view of relevant aspects (drops are the most obvious use case)

Grammar file: •

Can be modified to suite particular troubleshooting tasks • System will look for it at two locations: 1. 2.

•

disk0a:/usr/packet_drops.list /pkg/etc/packet_drops.list (default)

“show drops all commands” shows the constituent commands that will be called for parsing the final output

12

“Show drops all” sample output (1) RP/0/RP0/CPU0:ios#sh drops all commands Wed Feb 4 05:27:40.915 UTC Module CLI [arp] show arp traffic [cef] show cef drops [fabric] show controllers fabric fia drops egress [fabric] show controllers fabric fia drops ingress [lpts] show lpts pifib hardware entry statistics [lpts] show lpts pifib hardware police [lpts] show lpts pifib hardware static-police [netio] show netio drops [netio] fwd_netio_debug [niantic-driver] show controllers dmac client punt statistics [np] show controller np counters [np] show controllers np tm counters all [spp] show spp node-counters [spp] show spp client detail [spp] show spp ioctrl

13

“Show drops all” sample output (2) RP/0/RP0/CPU0:ios#sh drops all location 0/5/CPU0 Wed Feb 4 05:26:30.192 UTC ===================================== Checking for drops on 0/5/CPU0 ===================================== show cef drops: [cef:0/5/CPU0] Discard drops

packets : 5

show controllers fabric fia drops ingress: [fabric:FIA-0] sp0 crc err: 2746653 [fabric:FIA-0] sp0 bad align: 663 [fabric:FIA-0] sp0 bad code: 2 [fabric:FIA-0] sp0 align fail: 101 [fabric:FIA-3] sp1 prot err: 150577 show controller np counters: [np:NP0] MODIFY_PUNT_REASON_MISS_DROP: [np:NP1] MODIFY_PUNT_REASON_MISS_DROP: [np:NP2] MODIFY_PUNT_REASON_MISS_DROP: [np:NP3] PARSE_ING_DISCARD: 5 [np:NP3] PARSE_DROP_IN_UIDB_DOWN: 5 [np:NP3] MODIFY_PUNT_REASON_MISS_DROP:

1 1 1 1

14

netio CPU

SPP

Capturing lost packets in the NPU

3x 10G

Typhoon

3x10GE SFP +

3x 10G

Typhoon

3x10GE SFP +

FIA

3x 10G

CLI:

Typhoon 3x 10G

3x10GE SFP +

monitor np counter count

Typhoon 3x 10G

3x10GE SFP +

Typhoon

FIA

3x 10G

•

You can monitor any counter in the NPU on Typhoon generation line cards onwards

FIA

3x10GE SFP +

Typhoon

Typhoon

FIA

3x 10G 3x10GE SFP +

Typhoon

Captured packets are always dropped

•

Exists automatically after capturing packets or when timeout is reached

•

NPU is reset upon exit (~50ms forwarding stop)

•

Typhoon

3x 10G 3x10GE SFP +

•

•

Switch Fabric ASIC

•

3x10GE SFP +

This will be enhanced later

Packets subject to punt cannot be captured by this methodology »Use with care! 15

SPP packet captures RP/0/RSP0/CPU0#run attach 0/0/CPU0

RP/0/RSP0/CPU0:A9K-BNG#packet-trace spp platform protocol arp start-capture count 5 location 0/0/cpu0 Wed Mar 12 16:28:30.176 EDT Sending command: trace filter set 40 1 0x20 Trace filter set for protocol: ARP Sending command: trace start 5 Started capture for 5 packets

attach: Starting session 1 to node 0/0/CPU0 # spp_ui spp-ui> spp-ui> trace filter node client/punt Node "client/punt" set for trace filtering. Index: 11

spp-ui> trace filter set 52 4 0xD4000001 Modified filter for offset 52 successfully spp-ui> trace filter set 56 4 0xD4000002 Modified filter for offset 56 successfully

Wrote ASCII trace to /tmp/spp_packet_trace_ascii Sending command: trace print Packet serial 3 client/inject: length 110 phys_int_index -1 next_ctx 0xdeadbeef time 16:28:30.512 00: 00 65 7a 00 00 00 00 70 72 00 00 02 00 5e 00 00 10: 80 00 00 00 00 00 0f 8c 40 c1 0c c8 50 00 00 00 20: 00 00 0d 34 3f ff f2 90 20 04 fe 03 01 04 00 05 30: 00 00 00 00 5e 00 00 00 00 00 00 00 00 04 00 02 40: 40 00 10 34 ff ff ff ff ff ff 66 66 44 44 22 22

16

Decoder https://scripts.cisco.com/ui/use/xr_spp_ui_to_pcap

Understanding TCAM and ACL, Error messages, design and monitor

17

NP complex STATS MEMORY

What is a TCAM •

It is “reversed” memory:

LOOKUP MEMORY

Network Process Unit

-

FRAME MEMORY TCAM

•

Normal memory receives an address and provides the data on that location • TCAM receives a data pattern (aka key) and searches where that content is found • •

That is GREAT for matching! Deterministic performance!

•

But.. TCAM is power hungry, expensive and limited in size

•

ASR9000 uses TCAM for: •

VLAN matching (q or qiq combo matching to an EFP interface) • QOS class-map matching • ACL matching •

When A9K receives a packet a “key is built” and passed to tcam; this returns both ACL match results and qos class-map results. The key has a particular width (eg 144 or 576 bit) 18

QoS Classification Formats

• A given QoS policy-map generally classifies based on a single classification format • IPv4 and IPv6 classes can co-exist in the same policy Fields supported

Format 0

Format 1

Format 2

Format 3

Format 4

- IPV4 source address (Specific/Range)[1] - IPV4 Destination address (Specific/Range) - IPV4 protocol - IPV4 TTL - IPV4 Source port (Specific/Range) - IPV4 Destination port (Specific/Range) - TCP Flags - IP DSCP / TOS / Precedence - QOS-group (output policy only) - Discard-class (outputpolicy only) - EXP

- Outer VLAN/COS/DEI - Inner VLAN/COS - IPV4 Source address (Specific/Range) - IP DSCP / TOS / Precedence - QOS-group (output policy only) - Discard-class (output policy only) - EXP

- Outer VLAN/COS/DEI - Inner VLAN/COS - IPV4 Destination address (Specific/Range) - IP DSCP / TOS / Precedence - QOS-group (output policy only) - Discard-class (output policy only) - EXP

- Outer VLAN/COS/DEI - Inner VLAN/COS - MAC Destination address - MAC source address - QOS-group (output policy only) - Discard-class (output policy only)

- IPV6 source address (Specific/Range) - IPV6 Destination address (Specific/Range) - IPV6 protocol - IPV6 TOS /EXP - IPV6 TTL - IPV6 Source port (Specific/Range) - IPV6 Destination port (Specific/Range) - TCP Flags - Outer VLAN/COS/DEI - Inner VLAN/COS - IPV6 header flags - QOS-group (output policy only) - Discard-class (outputpolicy only)

[1]

All fields marked in green are defined using an ACL used for QOS classification.

ACL Architecture and Troubleshooting •

What is so special about ACL? • •

Challenges when scaling acl’s on TR cards To avoid security hole during in-place modification, XR uses make before break architecture (Just like Traffic-engineering re-optimization) •

But make-before-break architecture needs extra TCAM space during transition from old ACL to new modified ACL

The next set of slides go in detail to describe Different type of ACL’s and how to monitor TCAM usage • Troubleshooting and explanations for ACL modification failures • Available options to optimize TCAM usage •

20

Type of ACL’s •

Plain (Vanilla ACLs) •

Usually each ACE needs one TCAM entry. • To support in-place modification, 2X TCAM entries may need to be free. •

Example:

RP/0/RSP0/CPU0:ASR9K-1#show run ipv4 access-list acl-test ipv4 access-list acl-test 10 permit udp 104.193.122.128 0.0.0.127 any eq ntp 15 permit tcp host 67.132.144.133 host 67.132.144.134 range 1024 65535 20 permit ipv4 any any

P/0/RSP0/CPU0:ASR9K-1#show pfilter-ea feature ipv4-acl acl-test location 0/1/cpU0 Rgn acl-test, lkup v4, Dir Eg, Chan 1, acl_id 5, vmr_id 5, num_aces 4, num_tcam_entries 11, refcnt 1. ACE List for This Region: seq_num 10, tcam_entries 2, stats_ace_id 0x530652 (0x613290) new 0  Needs 2 TCAM entries seq_num 15, tcam_entries 7, stats_ace_id 0x530653 (0x613298) new 0  Needs 7 TCAM entries seq_num 20, tcam_entries 1, stats_ace_id 0x530654 (0x6132a0) new 0  Needs 1 TCAM entry seq_num 2147483647, tcam_entries 1, stats_ace_id 0x530655 (0x6132a8) new 0  Default deny deny entry Intf List for This Region: Te0/1/1/2, hw_count 0.  Applied to Tengig0/1/1/2 interface 21

Type of ACL’s •

Scale ACL (Uses object groups) •

Usually each ACE needs many TCAM entries • (To support in-place modification, 2X TCAM entries required) Example: object-group network ipv4 PROD 10.16.27.0/24 10.102.221.0/24 object-group network ipv4 net-group1 10.218.107.0/24 10.218.108.0/22

object-group port NFS_UDP eq sunrpc eq 635 object-group port range1 range 100 103

object-group network ipv4 h_prefix 192.168.162.140/32 object-group network ipv4 net-group2 192.168.39.0/26 192.168.40.0/26

RP/0/RSP0/CPU0:ASR9K-1#show run ipv4 access-list sample-acl 20 permit udp net-group net-group1 net-group net-group2 port-group range1 30 permit udp net-group PROD port-group NFS_UDP net-group h_prefix RP/0/RSP0/CPU0:ASR9K-1#show pfilter-ea fea ipv4-acl sample-acl location 0/1/cpu0 Rgn sample-acl, lkup v4, Dir Eg, Chan 1, acl_id 5, vmr_id 4, num_aces 3, num_tcam_entries 15, refcnt 1. ACE List for This Region: seq_num 20, tcam_entries 8, stats_ace_id 0x530650 (0x613280) new 0  Needs 8 TCAM entries seq_num 30, tcam_entries 6, stats_ace_id 0x530651 (0x613288) new 0  Needs 6 TCAM entries seq_num 2147483647, tcam_entries 1, stats_ace_id 0x530652 (0x613290) new 0 Intf List for This Region: Te0/1/1/2, hw_count 0.  Applied to interface Ten0/1/1/2 RP/0/RSP0/CPU0:ASR9K-1#

22

Scale ACL architecture •

Provides a mechanism to simplify ACL definitions and ability to perform compression

•

Uses object-groups for grouping the source/destination prefixes and source/destination UDP/TCP ports

object-group network ipv4 PROD_GFS 10.16.27.0/24 10.102.221.0/24 10.104.96.0/23 10.104.98.0/24 !

object-group port NFS_UDP eq sunrpc eq 635 eq 1110 eq 2049 range 38465 38467

object-group network ipv4 h_pefix 192.168.162.140/32 !

RP/0/RSP0/CPU0:ASR9K-1#show run ipv4 access-list sample-acl permit udp net-group PROD_GFS port-group NFS_UDP net-group h_prefix

Without scale ACL, ACL will look like below: permit udp permit udp permit udp permit udp permit udp permit udp permit udp permit udp permit udp permit udp ….. ---….. ---permit udp

10.16.27.0/24 eq sunrpc 192.168.162.140/32 10.16.27.0/24 eq 635 192.168.162.140/32 10.16.27.0/24 eq 635 192.168.162.140/32 10.16.27.0/24 eq 1110 192.168.162.140/32 10.16.27.0/24 eq 2049 192.168.162.140/32 10.16.27.0/24 eq 38465 192.168.162.140/32  port range expanded 10.16.27.0/24 eq 38466 192.168.162.140/32  port range expanded 10.16.27.0/24 eq 38467 192.168.162.140/32  port range expanded 10.102.221.0/24 eq sunrpc 192.168.162.140/32  above 8 entries repeat for 2nd prefix in PROD_GFS 10.102.221.0/24 eq 635 192.168.162.140/32 10.102.221.0/24 eq sunrpc 192.168.162.140/32  first 8 entries repeat for 3rd prefix in PROD_GFS

23

TCAM Allocation • Why single ACE may need more then one TCAM entry? • •

TCAM entries are represented in VMR (Value, Mask, Result) To represent a port we may need multiple VMR combinations

• Below ACL needs 32 TCAM entries: •

To cover all ports from 10-1023, we need 8 TCAM entries (below) • • • • • • • •

•

range_map[0][0].value:0xa range_map[0][1].value:0xc range_map[0][2].value:0x10 range_map[0][3].value:0x20 range_map[0][4].value:0x40 range_map[0][5].value:0x80 range_map[0][6].value:0x100 range_map[0][7].value:0x200

range_map[0][0].mask:0xfffe range_map[0][1].mask:0xfffc range_map[0][2].mask:0xfff0 range_map[0][3].mask:0xffe0 range_map[0][4].mask:0xffc0 range_map[0][5].mask:0xff80 range_map[0][6].mask:0xff00 range_map[0][7].mask:0xfe00

       

Covers Covers Covers Covers Covers Covers Covers Covers

ports ports ports ports ports ports ports ports

10-11 12-15 16-31 32-63 64-127 128-255 256-511 512-1023

Total TCAM entries needed: • 4 x 8 = 32 (4 combination of prefixes along with port range)

object-group network ipv4 net-group1 10.218.107.0/24 1 10.218.108.0/22 2

object-group network ipv4 net-group2 192.168.39.0/26 3 192.168.40.0/26 4

RP/0/RSP0/CPU0:ASR9K-1#show run ipv4 access-list sample-acl 20 permit udp net-group net-group1 net-group net-group2 port-group range1

object-group port range1 range 1024 65535 ! 24

ACL in-line modification • ACL architecture is built to suport in-line modification of ACL’s • In Line Modification of plain ACL’s without sequence-number (Example, loading config from the tftp-server using load or copy option) • ALL new entries programmed in TCAM • All entries following the new entries will be pushed down (hence will have new seq number and hence reprogramming) • Dynamic insertion of new rules

• In Line Modification of scale ACL • • •

To avoid security hole during ACL modification, a new copy of the ACL is programmed in TCAM Once new copy is programmed, pointer is moved from old copy to new copy in TCAM and old copy is deleted Note: Having shared object-groups between ACL’s will result in in-line modification of all ACL’s using THAT object-groups

•

In-line Modification of ACL from the router config mode: • Requires sequence number specified. Hence this does not require 2x TCAM entries during programming 25

ACL in-line modification [Example] • In Line Modification of plain ACL’s without sequence-number (Example, loading config from the tftp-server using load option) • ALL new entries gets programmed in TCAM (dark blue in below example) • All entries following the new entries will be pushed down (blue in below example) • Once programming is successful, old entries are removed (Example seq 20, 30 and 40)

ORIGNAL

CHANGES

Example 1: (Existing ACL) ipv4 access-list NCS 10 permit ipv4 10.33.0.0/16 any 20 permit ipv4 10.35.0.0/16 any 30 permit ipv4 10.36.0.0/16 any 40 permit ipv4 82.222.32.0/24 any interface gigabit 0/0/0/1 ipv4 access-group NCS egress

ACL File : (no sequence numbers) more acl_new  File in tftpserver ipv4 access-list NCS permit ipv4 10.33.0.0/16 any permit ipv4 10.34.0.0/16 any  new permit ipv4 10.35.0.0/16 any  pushed down permit ipv4 10.36.0.0/16 any  pushed down permit ipv4 82.222.32.0/24 any  pushed down

NEW ipv4 access-list NCS 10 permit ipv4 10.33.0.0/16 any 20 permit ipv4 10.34.0.0/16 any 30 permit ipv4 10.35.0.0/16 any 40 permit ipv4 10.36.0.0/16 any 50 permit ipv4 82.222.32.0/24 any

Perform in-line modification Router#copy file running-config OR Router-config#load 26

Scale ACL Compression • Scale ACL’s can be compressed to minimize TCAM usage • •

Less amount of TCAM’s used, but due to additional lookups needed, affect NP performance The object groups/lists are in search memory now and in tcam the entry is matching against the list entry: 1. SEARCH: “Find source/dest ip address belongs to what list? 2. TCAM: “If _that_list_X and port Y then match ACE”

• Typhoon Supports 3 compression levels: • Level 0: No compression. (highest performance) • •

• •

Simply expands the object-groups and creates ACE’s. Performance same as legacy ACLs Benefit: ACL definitions are very simple and can be grouped

Level 1: Source prefix compression. (medium pps performance) • Uses less amount of TCAM entries • Leads to NP performance hit. Level 3: Compresses all 4 parameters (source/dest IP and port numbers) (lowest pps performance)

• •

Further improves TCAM space usage Uses larger size TCAM entries (640 size Keys instead of 160 size keys. Hence utilizing TCAM space 27 meant for IPv6 ACL’s!!)

TCAM capacity ACL’s are programmed in NPU TCAM memory . Each NP has it’s own TCAM space

•

•

•

TCAM entries are shared between QOS, netflow, LPTS and ACL

Use below “show” command to check available TCAM space: RP/0/RSP0/CPU0:ASR9K-1#show prm server tcam summary all all np0 location 0/1/cpu0  SE card TCAM Logical Table: TCAM_LT_ODS2 (2), free entries: 89711, resvd 128  ODS2 == 160 bit TCAM keys TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 15205, resvd 128  ODS8 == 640 bit TCAM keys RP/0/RSP0/CPU0:ASR9K-1# 160 bit keys are used by : IPv4 applications and compression level 1 ACLs 640 bit keys are used by : IPV6 applications and compression level 3 ACLs

Description

TR cards

SE cards

comment

1

Total TCAM capacity (TCAM Key size : 160 bits)

24K

96K

• • •

Used by IPv4 Shared between ACL, netflow, LPTS, QOS 8K TCAM entries are allocated to internal use by default. Hence TCAM entries available for programming of ACL/netflow/LPTS/QOS are 8K less then the max (Eg: 16K for TR and 88K for SE cards)

2

Total TCAM capacity (TCAM key size : 640 bit)

4K

16K

• •

Used by IPv6 applications and compression level-3 ACL’s Shared between ACL, netflow, LPTS and QOS

28

Can my ACL fit into TCAM? •

Step 1: First check how much space is available in TCAM

RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#show prm server tcam summary all all np0 location 0/1/cpu0  TR card Node: 0/1/CPU0: TCAM summary for NP0: TCAM Logical Table: TCAM_LT_L2 (1) Partition ID: 0, priority: 2, valid entries: 21, free entries: 299 Partition ID: 1, priority: 2, valid entries: 0, free entries: 320 Partition ID: 2, priority: 1, valid entries: 0, free entries: 320 Partition ID: 3, priority: 1, valid entries: 52, free entries: 11788 Partition ID: 4, priority: 0, valid entries: 5, free entries: 11771 TCAM Logical Table: TCAM_LT_ODS2 (2), free entries: 15937, resvd 128  160 key size TCAM entries available Application ID: NP_APP_ID_ACL (2) Total: 3 vmr_ids, 500 active entries, 500 allocated entries.  TCAMs used by ACL (including compression level 0 and 1) TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 3365, resvd 128 Application ID: NP_APP_ID_ACL (2) Total: 0 vmr_ids, 0 active entries, 0 allocated entries.

 Number of 640 key size TCAMs available  TCAMs used by ACL (IPV6 and Compression level-3)

• Step 2: Configure the ACL but do not apply the ACL to the interface. You would need to commit the ACL to the config though.

29

Can my ACL fit into TCAM? •

Step 3: Check how many entries needed for programming ACL at different compression levels •

In the below example, Without compression ACL needs 61K entries and with level 3 compression, only 3 entries

R0/RSP0/CPU0:EAST-PE-ASR9K-1#show access-lists ipv4 tcam-test-acl hardware ingress resource-check location 0/1/cpu0 < Rules (ACE) : 2 ACL compression level : 0  Compression level 0 Fields compressed : None TCAM Entries required : 61505  TCAM entries need for compression level 0 TCAM Key Width : 160 ---------------------------------------------------------Rules (ACE) : 2 ACL compression level : 1  Compression level 1 Fields compressed : SrcIP Prefix Entry required : 125 TCAM Entries required : 497  TCAM entries need for compression level 0 TCAM Key Width : 160 ……… ---------------------------------------------------------Rules (ACE) : 2 ACL compression level : 3  Compression Level 3 Fields compressed : SrcIP, DstIP, SrcPort, DstPort Prefix Entry required : 171 TCAM Entries required : 3  TCAM entries need for compression level 0. Only 3 entries needed instead of 61K entries without compression TCAM Key Width : 640  TCAM key is larger which are used by ipv6 ACL ……… RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#

30

TCAM carving [Modify partition memory percentage]

• Default carving for 160 and 640 TCAM keys is 60-40% • •

Total number of 160 (ods2) bit TCAM keys available are 60% of total Total number of 640 (ods8) bit TCAM keys available are 40% of total

• Example for TR card is below: RP/0/RSP0/CPU0:ASR9K-2#show prm server tcam partition all location 0/1/cpu0  TR card ---------------------------------------------------------------TCAM partition information: 1 ods2 blk = 2048 entries, 1 ods8 blk = 512 entries NP0 : tot-ods2-blks 11 [61% of ods2+ods8 blks], used-ods2-blks 3 [17% of ods2+ods8 blks] NP0 : tot-ods8-blks 7 [39% of ods2+ods8 blks], used-ods8-blks 1 [ 6% of ods2+ods8 blks]

RP/0/RSP0/CPU0:ASR9K-2#

• Modifying TCAM Carving: ODS2 or ODS8 1. Change the TCAM allocation: RP/0/RSP0/CPU0:ASR9K-2(admin-config)#hw-module profile tcam tcam-part-70-30  To 70-30% from default 60-40%

2. Reload the LC for config changes to take effect: RP/0/RSP0/CPU0:WEST-PE-ASR9K-2#hw-module location 0/1/cpu0 reload

3. Check the new TCAM carvings using below command: RP/0/RSP0/CPU0:WEST-PE-ASR9K-2#show prm server tcam partition all ---------------------------------------------------------------TCAM partition information: 1 ods2 blk = 2048 entries, 1 ods8 blk NP0 : tot-ods2-blks 13 [72% of ods2+ods8 blks], used-ods2-blks 3 NP0 : tot-ods8-blks 5 [28% of ods2+ods8 blks], used-ods8-blks 1

location 0/1/cpu0 = 512 entries [17% of ods2+ods8 blks] [ 6% of ods2+ods8 blks]

31

Common Issues and Error messages 32

How to Identify ACL out of sync

• •

Out of sync can cause ACL mis-programming in Hardware (TCAM) Check content of each ACE in the below output to verify there is no out of sync

RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#show run ipv4 access-list test-acl  config or sysdb view of the ACL ipv4 access-list test-acl 40 permit tcp any net-group n_10.0.0.0_8 established 70 permit udp net-group n_192.168.0.0_16 port-group KERBEROS_UDP net-group FB-NETS 230 deny ipv4 any any RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#show access-lists ipv4 test-acl usage pfilter location all Interface : GigabitEthernet0/2/0/7  ACL is applied to 0/2/0/7 which is in slot 0/2/cpu0 Input Common-ACL : N/A ACL : test-acl Output ACL : N/A

RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#describe show access-lists ipv4 test-acl usage pfilter location 0/2/cpu0 | begin spawn Spawn the process: ipv4_acl_usage_cmd -a test-acl -n 2081 -i 0x1  Node-id for a LC (here 0/2/cpu0). You need this info for next commands RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#show access-lists ipv4 test-acl  This is how ACL is stored in ACL Manager in RSP ipv4 access-list test-acl 40 permit tcp any net-group n_10.0.0.0_8 established 70 permit udp net-group n_192.168.0.0_16 port-group KERBEROS_UDP net-group FB-NETS 230 deny ipv4 any any RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#run ipv4_acl_show -p pfilter_ea -i 2081  This is the ACL pfilter_ea gets from ACL Manager ipv4 access-list test-acl 40 permit tcp any net-group n_10.0.0.0_8 established 70 permit udp net-group n_192.168.0.0_16 port-group KERBEROS_UDP net-group FB-NETS 230 deny ipv4 any any RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#run ipv4_acl_show -p netio -i 2081  This is the ACL netio gets from ACL manager ipv4 access-list test-acl 40 permit tcp any net-group n_10.0.0.0_8 established 70 permit udp net-group n_192.168.0.0_16 port-group KERBEROS_UDP net-group FB-NETS 230 deny ipv4 any any

33

Well known error messages: • Commit fails with following error message: % The process 'ipv4_acl_mgr' took too long to respond to a verification request and was timed out

• Meaning of the Error message: •Process ipv4_acl_mgr couldn't get response from low level process within timeout period

• What needs to be collected if above commit error is seen: •In-line modification of the ACL results in programming of TCAM. Flow of the information to TCAM is as follows: • Ipv4_acl_mgr  GSP  pfilter_ea  PRM  TCAM •Collect output of “show process blocked all” command to see which process in the above chain blocked. •Once blocking process is identified collect output of “follow job iteration 5 location •Other general commands

• ` 34

Error Message: !% The process 'ipv4_acl_mgr'ipv4_acl_mgr' took too long to respond to a verification request and was timed out

• Analysis of below output: • •

Below output shows that, ipv4_acl_mgr is blocked on GSP and GSP is blocked by pfilter_ea in 0/1/cpu0 pfilter_ea waiting for prm_server_ty to complete the task assigned (programming TCAM)

RP/0/RSP0/CPU0:WEST-PE-ASR9K-1#show process blocked location all node: node0_0_CPU0 -----------------------------------------------------------------Jid Pid Tid Name State TimeInState Blocked-on 297 172115 22 prm_server_ty Mutex 0:00:01:0438 172115-33 #1 288 467062 1 pifibm_server_lc Reply 0:00:00:0138 172115 prm_server_ty 285 516255 1 pfilter_ea Reply 0:00:00:0397 172115 prm_server_ty  pfilter_ea is blocked on prm_server_ty node: node0_RSP0_CPU0 -----------------------------------------------------------------287 258200 1 ipv4_acl_mgr Reply 0:00:47:0773 204882 gsp 6 pages) • Provide cheat sheet exposing install/upgrade shortcuts 9+ Improvements DDTS being delivered in XR5.3.2 for ASR9K/CRS

1st phase progressing as part of XR5.3.2

Marketing to communicate RSP2 EoS

Completed

Completed Executing as per plan

172

Summary of Install upgrade improvements

Upgrade instructions, cheat sheet and education • Step by step upgrade (Life support on RSP2) https://supportforums.cisco.com/document/145991/managing-disk-space-rsp-4grsp-8g-aka-rsp2

• Bridge SMUs on a diet • The size of the bridge SMU was 1.5G, its now 400M • General Upgrade instruction document was 40 pages • Shrunk 5.1.x, 5.2.x, 5.3.x to 6 pages http://www.cisco.com/web/Cisco_IOS_XR_Software/index.html

• Install cheat sheet… Exposing the install upgrade shortcuts https://supportforums.cisco.com/document/12440491/ios-xr-install-upgrade-cheat-sheet

174

Install enhancements • Support for full pie install handling •

Wrapper pie so a single file is used to install

• XR Ability to install more than 16 pies simultaneously •

Increase .tar file size limit

• Break the activate operation in two stages activate and reload •

Install ops is like a bullet train, hard to stop, with activate and hold the software is installed, but holds short of reload

• yum like installer for IOS XR •

Automatic dependency management

175

CSM Server - Capabilities 1)

Software Deployment Automation from any-to-any release or many-to-many releases

2)

It does Major release upgrade or Simple SMU provisioning Automation

3)

Pre upgrade, Upgrade and post upgrade verification

4)

Flexible Python does the upgrades, its open source and modifiable

5)

SW Inventory, tracking, SMU management, Software Download from CCO (Integrated CSMv2 Client)

6)

Supported on CRS-1, ASR9K! NCS6K under development

7)

Builds on what CSM Client does best today to provide you complete software automation 176

CSM Server hosting requirements 1)

UCS like or VM running Linux - 10g disk(depends on network scale), 8mb ram, 1 CPU

2)

Preferable Debian, Ubuntu & CentOS, redhat supported

3)

Need Internet access, installation takes 5-10 minutes

4)

Other Software will be downloaded like, pip, python 2.7, mysql etc..

Deployment Model •

CSM server through Jump Server, Multihop access

•

CSM server on Jump Server

•

CSM server through Console or Terminal Server for Aux/Con Installations

•

CSM server Direct

Thank you

179

Backup 180

181

182

183

184

Participate in the “My Favorite Speaker” Contest Promote Your Favorite Speaker and You Could Be a Winner •

Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

•

Send a tweet and include Your favorite speaker’s Twitter handle • Two hashtags: #CLUS #MyFavoriteSpeaker •

•

You can submit an entry for more than one of your “favorite” speakers

•

Don’t forget to follow @CiscoLive and @CiscoPress

•

View the official rules at http://bit.ly/CLUSwin

185

Complete Your Online Session Evaluation •

Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

•

Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect. Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 186

Continue Your Education •

Demos in the Cisco campus

•

Walk-in Self-Paced Labs

•

Table Topics

•

Meet the Engineer 1:1 meetings

•

Related sessions

187

Internet of Things (IoT) Cisco Education Offerings Course

Description

Cisco Certification

NEW! CCNA Industrial

An associate level instructor led training course designed to prepare you for the CCNA Industrial certification

CCNA® Industrial

Managing Industrial Networks with Cisco Networking Technologies (IMINS)

This curriculum addresses foundational skills needed to manage and administer networked industrial control systems. It provides plant administrators, control system engineers and traditional network engineers with an understanding of the networking technologies needed in today's connected plants and enterprises

Cisco Industrial Networking Specialist

Control Systems Fundamentals for Industrial Networking (ICINS)

For IT and Network Engineers, covers basic concepts in Industrial Control systems including an introduction to automation industry verticals, automation environment and an overview of industrial control networks

Networking Fundamentals for Industrial Control Systems (INICS)

For Industrial Engineers and Control System Technicians, covers basic IP and networking concepts, and introductory overview of Automation industry Protocols.

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 189

Business Transformation Cisco Education Offerings Course

Description

Cisco Certification

For IT and Network Professionals: Building Business Specialist Skills

•

Builds non-technical skills key to ensure business impact and influence. Topics include: business analysis, finance, technology adoption and effective communications.

•

Bridges IT and business impacts of mature and emerging solutions including cloud plus Internet of Everything

Cisco Enterprise IT Business Specialist

For Technology Sellers: Applying Cisco Specialized Business Value Analysis Skills

Builds skills to discover and address technology needs using a businessfocused, consultative sales approach

Cisco Business Value Specialist

Executing Advanced Cisco Business Value Analysis and Design Techniques

Enables customer transformation through business architecture and solution selling expertise

Cisco Certified Business Value Practitioner

Performing Cisco Business-Focused Transformative Architecture Engagements

Provides skills and an approach to build a strategic roadmap of IT initiatives, aligned to business priorities

Cisco Transformative Architecture Specialist

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 190

Security Cisco Education Offerings Course Implementing Cisco IOS Network Security (IINS) Implementing Cisco Edge Network Security Solutions (SENSS)

Description Focuses on the design, implementation, and monitoring of a comprehensive security policy, using Cisco IOS security features

Cisco Certification CCNA® Security

Configure Cisco perimeter edge security solutions utilizing Cisco Switches, Cisco Routers, and Cisco Adaptive Security Appliance (ASA) Firewalls

Implementing Cisco Threat Control Solutions (SITCS) Implementing Cisco Secure Access Solutions (SISAS)

Implementing Cisco Secure Mobility Solutions (SIMOS)

Deploy Cisco’s Next Generation Firewall (NGFW) as well as Web Security, Email Security and Cloud Web Security

Deploy Cisco’s Identity Services Engine and 802.1X secure network access Protect data traversing a public or shared infrastructure such as the Internet by implementing and maintaining Cisco VPN solutions

Securing Cisco Networks with Threat Detection and Analysis (SCYBER)

Designed for professional security analysts, the course covers essential areas of competency including event monitoring, security event/alarm/traffic analysis, and incident response

Network Security Product and Solutions Training

For official product training on Cisco’s latest security products, including Adaptive Security Appliances, NGIPS, Advanced Malware Protection, Identity Services Engine, Email and Web Security Appliances see www.cisco.com/go/securitytraining

Cisco Cybersecurity Specialist

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 191

R&S Related Cisco Education Offerings Course

Description

Cisco Certification

CCIE R&S Advanced Workshops (CIERS-1 & CIERS-2) plus Self Assessments, Workbooks & Labs

Expert level trainings including: instructor led workshops, self assessments, practice labs and CCIE Lab Builder to prepare candidates for the CCIE R&S practical exam.

CCIE® Routing & Switching

• Implementing Cisco IP Routing v2.0 • Implementing Cisco IP Switched Networks V2.0 • Troubleshooting and Maintaining Cisco IP Networks v2.0

Professional level instructor led trainings to prepare candidates for the CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in self study eLearning formats with Cisco Learning Labs.

CCNP® Routing & Switching

Interconnecting Cisco Networking Devices: Part 2 (or combined)

Configure, implement and troubleshoot local and wide-area IPv4 and IPv6 networks. Also available in self study eLearning format with Cisco Learning Lab.

CCNA® Routing & Switching

Interconnecting Cisco Networking Devices: Part 1

Installation, configuration, and basic support of a branch network. Also available in self study eLearning format with Cisco Learning Lab.

CCENT® Routing & Switching

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 192

Wireless Cisco Education Offerings Course

Description

Cisco Certification

• Conducting Cisco Unified Wireless Site Survey • Implementing Cisco Unified Wireless Voice Networks • Implementing Cisco Unified Wireless Mobility Services • Implementing Cisco Unified Wireless Security Services

Professional level instructor led trainings to prepare candidates to conduct site surveys, implement, configure and support APs and controllers in converged Enterprise networks. Focused on 802.11 and related technologies to deploy voice networks, mobility services, and wireless security.

CCNP® Wireless

Implementing Cisco Unified Wireless Network Essential

Prepares candidates to design, install, configure, monitor and conduct basic troubleshooting tasks of a Cisco WLAN in Enterprise installations.

CCNA® Wireless

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 193

Design Cisco Education Offerings Course

Description

Cisco Certification

Designing Cisco Network Service Architectures (ARCH)

Provides learner with the ability to perform conceptual, intermediate, and detailed design of a network infrastructure that supports desired capacity, performance, availability required for converged Enterprise network services and applications.

CCDP® (Design Professional)

Designing for Cisco Internetwork Solutions (DESGN)

Instructor led training focused on fundamental design methodologies used to determine requirements for network performance, security, voice, and wireless solutions. Prepares candidates for the CCDA certification exam.

CCDA® (Design Associate)

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 194

Service Provider Cisco Education Offerings Course Deploying Cisco Service Provider Network Routing (SPROUTE) & Advanced (SPADVROUTE) Implementing Cisco Service Provider Next-Generation Core Network Services (SPCORE)

Description SPROUTE covers the implementation of routing protocols (OSPF, IS-IS, BGP), route manipulations, and HA routing features; SPADVROUTE covers advanced routing topics in BGP, multicast services including PIM-SM, and IPv6;

Cisco Certification CCNP Service Provider®

SPCORE covers network services, including MPLS-LDP, MPLS traffic engineering, QoS mechanisms, and transport technologies;

Edge Network Services (SPEDGE)

SPEDGE covers network services, including MPLS Layer 3 VPNs, Layer 2 VPNs, and Carrier Ethernet services; all within SP IP NGN environments.

Building Cisco Service Provider Next-Generation Networks, Part 1&2 (SPNGN1), (SPNGN2)

The two courses introduce networking technologies and solutions, including OSI and TCP/IP models, IPv4/v6, switching, routing, transport types, security, network management, and Cisco OS (IOS and IOS XR).

CCNA Service Provider®

Implementing Cisco Service Provider Mobility UMTS Networks (SPUMTS); Implementing Cisco Service Provider Mobility CDMA Networks (SPCDMA); Implementing Cisco Service Provider Mobility LTE Networks (SPLTE)

The three courses (SPUMTS, SPCDMA, SPLTE) cover knowledge and skills required to understand products, technologies, and architectures that are found in Universal Mobile Telecommunications Systems (UMTS) and Code Division Multiple Access (CDMA) packet core networks, plus their migration to Long-Term Evolution (LTE) Evolved Packet Systems (EPS), including Evolved Packet Core (EPC) and Radio Access Networks (RANs).

Cisco Service Provider Mobility CDMA to LTE Specialist; Cisco Service Provider Mobility UMTS to LTE Specialist

Implementing and Maintaining Cisco Technologies Using IOS XR (IMTXR)

Service Provider/Enterprise engineers to implement, verification-test, and optimize core/edge technologies in a Cisco IOS XR environment.

Cisco IOS XR Specialist

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 195

Collaboration Cisco Education Offerings Course

Description

Cisco Certification

CCIE Collaboration Advanced Workshop (CIEC)

Gain expert-level skills to integrate, configure, and troubleshoot complex collaboration networks

CCIE® Collaboration

Implementing Cisco Collaboration Applications (CAPPS)

Understand how to implement the full suite of Cisco collaboration applications including Jabber, Cisco Unified IM and Presence, and Cisco Unity Connection.

CCNP® Collaboration

Implementing Cisco IP Telephony and Video Part 1 (CIPTV1)

Learn how to implement Cisco Unified Communications Manager, CUBE, and audio and videoconferences in a single-site voice and video network.

CCNP® Collaboration

Implementing Cisco IP Telephony and Video Part 2 (CIPTV2)

Obtain the skills to implement Cisco Unified Communications Manager in a modern, multisite collaboration environment.

Troubleshooting Cisco IP Telephony and Video (CTCOLLAB)

Troubleshoot complex integrated voice and video infrastructures

Implementing Cisco Collaboration Devices (CICD)

Acquire a basic understanding of collaboration technologies like Cisco Call Manager and Cisco Unified Communications Manager.

Implementing Cisco Video Network Devices (CIVND)

Learn how to evaluate requirements for video deployments, and implement Cisco Collaboration endpoints in converged Cisco infrastructures.

CCNA® Collaboration

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 196

Data Center / Virtualization Cisco Education Offerings Course

Description

Cisco Certification

Cisco Data Center CCIE Unified Fabric Workshop (DCXUF); Cisco Data Center CCIE Unified Computing Workshop (DCXUC)

Prepare for your CCIE Data Center practical exam with hands on lab exercises running on a dedicated comprehensive topology

CCIE® Data Center

Implementing Cisco Data Center Unified Fabric (DCUFI); Implementing Cisco Data Center Unified Computing (DCUCI)

Obtain the skills to deploy complex virtualized Data Center Fabric and Computing environments with Nexus and Cisco UCS.

CCNP® Data Center

Introducing Cisco Data Center Networking (DCICN); Introducing Cisco Data Center Technologies (DCICT)

Learn basic data center technologies and how to build a data center infrastructure.

CCNA® Data Center

Product Training Portfolio: DCAC9k, DCINX9k, DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K

Get a deep understanding of the Cisco data center product line including the Cisco Nexus9K in ACI and NexusOS modes

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 197

Network Programmability Cisco Education Offerings Course

Description

Cisco Certification

Integrating Business Applications with Network Programmability (NIPBA); Integrating Business Applications with Network Programmability for Cisco ACI (NPIBAACI)

Learn networking concepts, and how to deploy and troubleshoot programmable network architectures with these self-paced courses.

Cisco Business Application Engineer Specialist Certification

Developing with Cisco Network Programmability (NPDEV); Developing with Cisco Network Programmability for Cisco ACI (NPDEVACI)

Learn how to build applications for network environments and effectively bridge the gap between IT professionals and software developers.

Cisco Network Programmability Developer Specialist Certification

Designing with Cisco Network Programmability (NPDES); Designing with Cisco Network Programmability for Cisco ACI (NPDESACI)

Learn how to expand your skill set from traditional IT infrastructure to application integration through programmability.

Cisco Network Programmability Design Specialist Certification

Implementing Cisco Network Programmability (NPENG); Implementing Cisco Network Programmability for Cisco ACI (NPENGACI)

Learn how to implement and troubleshoot open IT infrastructure technologies.

Cisco Network Programmability Engineer Specialist Certification

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 198

Cloud Cisco Education Offerings Course

Description

Designing the FlexPod Solution (FPDESIGN); Implementing and Administering the FlexPod Solution (FPIMPADM)

Learn how to design, implement and administer FlexPod solutions

UCS Director (UCSDF)

Learn how to manage physical and virtual infrastructure using orchestration and automation functions of UCS Director.

Cisco Prime Service Catalog

Learn how to deliver data center, workplace, and application services in an on-demand, automated, and repeatable method.

Cisco Intercloud Fabric

Learn how to implement end-to-end hybrid clouds with Intercloud Fabric for Business and Intercloud Fabric for Providers.

Cisco Intelligent Automation for Cloud

Learn how to implement and manage cloud deployments with Cisco Intelligent Automation for Cloud

Cisco Certification FlexPod Design Specialist; FlexPod Implementation & Administration Specialist

For more details, please visit: http://learningnetwork.cisco.com Questions? Visit the Learning@Cisco Booth or contact [email protected] 199