Introduction

Introduction to Realtimepublishers by Don Jones, Series Editor

For several years, now, Realtime has produced dozens and dozens of high-quality books that just happen to be delivered in electronic format—at no cost to you, the reader. We’ve made this unique publishing model work through the generous support and cooperation of our sponsors, who agree to bear each book’s production expenses for the benefit of our readers. Although we’ve always offered our publications to you for free, don’t think for a moment that quality is anything less than our top priority. My job is to make sure that our books are as good as—and in most cases better than—any printed book that would cost you $40 or more. Our electronic publishing model offers several advantages over printed books: You receive chapters literally as fast as our authors produce them (hence the “realtime” aspect of our model), and we can update chapters to reflect the latest changes in technology. I want to point out that our books are by no means paid advertisements or white papers. We’re an independent publishing company, and an important aspect of my job is to make sure that our authors are free to voice their expertise and opinions without reservation or restriction. We maintain complete editorial control of our publications, and I’m proud that we’ve produced so many quality books over the past years. I want to extend an invitation to visit us at http://nexus.realtimepublishers.com, especially if you’ve received this publication from a friend or colleague. We have a wide variety of additional books on a range of topics, and you’re sure to find something that’s of interest to you—and it won’t cost you a thing. We hope you’ll continue to come to Realtime for your educational needs far into the future. Until then, enjoy. Don Jones

i

Table of Contents Introduction to Realtimepublishers.................................................................................................. i Chapter 1: Forward…Into the Past!.................................................................................................1 VoIP vs. IPT.....................................................................................................................................3 The Uni-Media Phase ......................................................................................................................3 The Multi-Media Phase....................................................................................................................4 VoIP .....................................................................................................................................5 Technical Definition vs. Market Definition.....................................................................................5 VoIP is Revolutionary......................................................................................................................6 VoIP Business Aspects Not Clear....................................................................................................7 IPT........................................................................................................................................9 Major Changes ...............................................................................................................................12 Telephony Becomes an Application ..................................................................................13 Responsibility Shifts from Carrier to End-User Organizations .........................................14 The IPT Life Cycle ........................................................................................................................15 Planning and Assessment...................................................................................................16 Pre-Deployment Testing and Implementation ...................................................................17 Ongoing Operations and Optimization ..............................................................................17 Optimization ......................................................................................................................18 Summary ........................................................................................................................................19 Chapter 2: The IP Telephony Life Cycle.......................................................................................20 The Bigger Business Picture..........................................................................................................20 Packet Telephony Is Inevitable..........................................................................................21 Impact of VoP ....................................................................................................................27 VoIP and IPT Return on Effort..........................................................................................28 Enterprise Telephony Migration ....................................................................................................28 The Budget Process............................................................................................................28 Reuse of Existing Assets........................................................................................29 New Assets Needed ...............................................................................................30 Assessing Urgency and Prioritizing Migration..................................................................31 Migration............................................................................................................................31 Avoiding Migration ...............................................................................................31 Partial Migration ....................................................................................................32 Full Migration ........................................................................................................32

ii

Table of Contents Delaying VoIP/IPT ............................................................................................................32 Systems Integrator vs. Do It Yourself................................................................................32 Managed Service vs. Do It Yourself..................................................................................33 SLAs ..................................................................................................................................34 Penalties .................................................................................................................34 True Impact of SLA on Business and Operations .................................................35 The IPT Life Cycle ........................................................................................................................35 Planning and Assessment...................................................................................................36 Pre-Deployment Testing and Implementation ...................................................................37 Ongoing Operations and Optimization ..............................................................................37 Managing the IPT Life Cycle ........................................................................................................38 Enterprise-Accessible Tools ..............................................................................................38 MSP-Provided Customer Network Management ..................................................38 Manufacturer-Provided Systems............................................................................38 Third-Party Tools...................................................................................................39 Importance of Modeling, Monitoring, Measuring, and Managing ........................39 Summary ........................................................................................................................................40 Chapter 3: Planning and Assessment.............................................................................................41 Business Drivers for Change .........................................................................................................42 Lower Costs .......................................................................................................................44 The Myth of Bandwidth Savings ...........................................................................45 Access Charge/Local .............................................................................................46 Long Distance ........................................................................................................46 Support Costs .........................................................................................................47 Remote/Nomadic/Mobile Workers........................................................................48 Enabling New Strategic Applications................................................................................48 Worker Productivity...........................................................................................................49 Obsolescence of Older System ..........................................................................................49 No Longer Want to Invest in Old Technology ......................................................50 Manufacturer Discontinuation of Product and/or Support.....................................50 Increased Support and Maintenance Costs ............................................................50 Lack of Expansion Capability of Old System........................................................51 Setting Proper Expectations...........................................................................................................51

iii

Table of Contents Management Expectations .................................................................................................51 End-User Expectations.......................................................................................................52 Inventory of Existing Capabilities .................................................................................................53 Existing Telephony Performance Expectations Audit.......................................................54 Existing Telephony Budgeting, Relationships and Tools Audit........................................55 Existing Capabilities Audit ................................................................................................55 Synchronizing Existing and New Capabilities ..................................................................58 Network Readiness Assessment ....................................................................................................59 The Network Readiness Assessment Process ....................................................................61 In a Perfect World..................................................................................................62 In the Real World...................................................................................................62 Understanding the Key Metrics of a Network Assessment ...............................................62 High Availability ...................................................................................................64 Call and Voice Quality...........................................................................................65 Tips and Tricks for A Successful Network Assessment....................................................71 Simulating Calling Patterns ...................................................................................72 Simulating Gateways .............................................................................................72 Plan For The Future ...............................................................................................74 Assessment Tools...........................................................................................................................74 Third-Party Tools...............................................................................................................74 Third-Party Tools Selection “Report Card”...........................................................75 Summary: The Road to Success ....................................................................................................76 Chapter 4: Design and Pre-Deployment Testing ...........................................................................78 Network Design Considerations ....................................................................................................79 The Team ...........................................................................................................................79 Assumptions.......................................................................................................................83 The Voice/Data/Video “Triple Play” and Unified Communications ................................84 Real-Time ..............................................................................................................85 Near Real-Time......................................................................................................86 Background/Batch..................................................................................................86 Interrelationship of Multi-Media Services.........................................................................86 Application Availability.........................................................................................86 Delivery/Accuracy/Loss ........................................................................................90

iv

Table of Contents Delay ......................................................................................................................92 Delay Variation or Jitter.........................................................................................92 Differentiation, Prioritization and Queue Management.........................................93 Voice QoS..........................................................................................................................95 QoE ........................................................................................................................96 QoE vs. QoS...........................................................................................................96 Voice Metrics and Measurements........................................................................101 Impact on Business Processes..........................................................................................103 Standardizing Positive Impacts............................................................................104 Avoiding Negative Impacts .................................................................................104 Translating Needs to SLAs ..........................................................................................................104 The “Proper” SLA............................................................................................................105 Metrics .............................................................................................................................105 Measurement....................................................................................................................106 Establishing Feedback Loops and Review Cycles...........................................................106 Validation of Design ....................................................................................................................107 Network Tuning ...............................................................................................................107 Test Bed Architecture ......................................................................................................108 Network Impairments ..........................................................................................108 Timing/Synchronization Issues............................................................................109 Broadband Bandwidth Measurement and Calculation ........................................110 Gateway, Softswitch, Session Border Controller and IP Device Capacity and Performance .........................................................................................................111 Testing, Feedback, Network Modification, Testing Loop...........................................................112 Testing/Proof of Concept Objectives...................................................................113 Feature Support and End-to-End Operation ........................................................113 Closed Lab Environment .....................................................................................115 After Hours/Off Hours Familiarization and Benchmarking ................................115 Busy Hour Testing in a Live Network.................................................................115 Evaluation of Results and Ensuring End-User Acceptance.................................115 Fall-Back and Contingency Planning ..................................................................118 Use of Third-Party Tools .................................................................................................118 Summary ......................................................................................................................................119 Chapter 5: Implementation and Migration...................................................................................120 v

Table of Contents External Issues .............................................................................................................................120 Reducing Negative Impact of Implementation/Migration...............................................120 Business Cycles ...............................................................................................................121 Holidays ...........................................................................................................................122 Coordination ....................................................................................................................122 Plans.............................................................................................................................................123 Implementation Plan ........................................................................................................125 Training Plan....................................................................................................................125 Test/Acceptance Plan.......................................................................................................126 Migration Plan .................................................................................................................126 Ongoing Operations Plan.................................................................................................127 Contingency and Business/System Continuity Plan ........................................................127 Contingency Plan .................................................................................................128 Business/System Continuity Plan ........................................................................128 Escalation Procedure........................................................................................................129 Internal Issues ..............................................................................................................................129 Creation of Baseline and User Profiles............................................................................129 Bringing All Users to Baseline ........................................................................................130 Additional Capabilities by User Profile...........................................................................130 Security Issues .................................................................................................................130 Liaison with Security Department .......................................................................131 Firewalls and Proxy Server Issues .......................................................................131 IP Addresses and Port Numbers ..........................................................................132 NAT and Internal and External IP Addresses......................................................133 Internal and External Phone Number Mapping ...............................................................133 Flash Cut vs. Parallel Operation ......................................................................................134 Migration Priority and Order ...........................................................................................134 Involvement of End Users in Acceptance........................................................................134 IP Telephony Implementation and Migration Case Study...........................................................135 Business ...........................................................................................................................137 ROI and TCO.......................................................................................................137 Establish Site Profiles ..........................................................................................138 Rough-Order-of-Magnitude Pricing ....................................................................139

vi

Table of Contents Management Review and Site Prioritization .......................................................139 Initial Migration/Green Field Plan and Renegotiation of Favorable Contracts...139 Joint Migration/Green Field Planning Effort.......................................................139 Contracts and Delivery ........................................................................................140 Inventory and Fixed Asset Accounting and Disposition of Old Equipment .......140 Implementation of Changes to Business Operations ...........................................140 Technical..........................................................................................................................142 Review of Standards’ Status and Maturity ..........................................................142 Develop Private Net, IP VPN, and Carrier/PTT Strategies .................................142 Private Backbone Design/Upsize for Voice Traffic and Backbone Re-engineering143 Establish Common Standards and Test Plan .......................................................143 Establish Private Net, IP VPN, and Carrier/PTT Demo/Test Capabilities ..........143 Training/Staging for Migration Teams ................................................................143 Training for Support Teams.................................................................................144 Review of Planning and New Telephony Project Rollouts .................................144 Legal ................................................................................................................................144 Closing Contracts for Old Telephony System .....................................................144 New Contracts......................................................................................................144 Regulatory........................................................................................................................145 Compliance with National Law and Telco/PTT Regulation................................145 Contact Centers/Call Centers...........................................................................................146 Special Considerations and Contingency Planning .............................................146 Phased Implementation and Call-Taker Training ................................................146 Summary ......................................................................................................................................146 Chapter 6: Ongoing Operations ...................................................................................................147 Network Operation.......................................................................................................................147 Capacity, Performance, and the Broadband Flip .............................................................148 Reliability.............................................................................................................156 Security ................................................................................................................156 Call Quality and Voice Quality............................................................................160 Measurement and Monitoring..............................................................................163 Growth Management ...........................................................................................164 Recordkeeping and Documentation.................................................................................166

vii

Table of Contents Fixed Asset Accounting.......................................................................................166 Shipping, Warehousing, and Tracking.................................................................167 Repair/Refurbishment/Return to Service/End-of-Life.........................................167 Administration .............................................................................................................................168 End-User Cost Allocation ................................................................................................168 Detailed Usage and Billing ..............................................................................................168 Vendor/MSP/Carrier Billing............................................................................................169 Cost Reduction and Network Utilization Maximization .................................................169 Provisioning .................................................................................................................................169 Maintenance.................................................................................................................................170 Moves, Adds, and Changes..............................................................................................170 Technology Refresh .........................................................................................................170 Audits...............................................................................................................................170 Operations Tools..........................................................................................................................171 Integration ........................................................................................................................171 Operation..........................................................................................................................171 Summary ......................................................................................................................................171 Chapter 7: Optimization...............................................................................................................172 SLAs and the Scope of Optimization...........................................................................................172 The Optimization Cycle...............................................................................................................173 The Use of Software Tools ..........................................................................................................174 Tool Ubiquity...................................................................................................................174 Constant Testing and Monitoring ....................................................................................175 Exception Reporting and Filtering...............................................................................................175 “What If” Scenarios .........................................................................................................175 Trending and Capacity Planning..................................................................................................176 Bandwidth ........................................................................................................................176 Ports/Lines .......................................................................................................................177 Phone Numbers/Numbering Plan ....................................................................................179 Grooming .........................................................................................................................180 Gateways..............................................................................................................181 TDM Grooming ...................................................................................................181 VLAN Capacity ...............................................................................................................182

viii

Table of Contents VPN Capacity ..................................................................................................................182 Infrastructure....................................................................................................................182 Hardware......................................................................................................................................182 Repair...............................................................................................................................183 Obsolescence/Refresh ......................................................................................................184 Software .......................................................................................................................................184 Maintenance.....................................................................................................................184 Enhancements ..................................................................................................................185 Upgrades ..........................................................................................................................185 Patches and Security Audits.............................................................................................185 SLA Improvements......................................................................................................................186 Delay ................................................................................................................................186 Call Setup.............................................................................................................187 During Call...........................................................................................................187 Call Tear Down/Release ......................................................................................187 Delay Variation................................................................................................................187 Sample and Packet Loss...................................................................................................188 Availability ......................................................................................................................188 Telephony Service Availability ...................................................................................................189 GoS ..................................................................................................................................189 Blocking/Non Blocking Access...........................................................................190 Success Rates of Call Setup.................................................................................191 Related Gateway Issues .......................................................................................191 Prices and Costs ...........................................................................................................................191 Hardware Costs................................................................................................................191 Service and Support Costs ...............................................................................................192 In-Sourcing vs. Outsourcing ............................................................................................193 IP Contact Center Optimization...................................................................................................193 Summary ......................................................................................................................................194 Chapter 8: Sweating the Details and Assuring Success...............................................................195 Legal and Regulatory Issues ........................................................................................................195 General Considerations....................................................................................................196 Countrywide Bans of VoIP Services or Service Providers..................................197

ix

Table of Contents Penalties for Non-Compliance.............................................................................197 Emergency Services.............................................................................................198 Security and Wiretapping/Call Recording ...........................................................198 Records Retention................................................................................................199 Taxes and Fees.....................................................................................................199 Country-Specific Regulatory Issues ................................................................................199 Contractual Issues ............................................................................................................199 Patent Issues.....................................................................................................................200 Technical Issues ...........................................................................................................................201 Voice Band Data ..............................................................................................................201 Numbering Plans..............................................................................................................202 Voice VPNs .........................................................................................................202 E.164 Compliance................................................................................................202 URIs/URLs ..........................................................................................................203 ENUM and Other Numbering Issues...................................................................204 Interworking Issues with IPT Service Providers .............................................................204 The New Demark.................................................................................................204 SIP-T ....................................................................................................................208 Electrical Power ...................................................................................................209 Business Issues.............................................................................................................................209 Budgets ............................................................................................................................210 Staff......................................................................................................................210 Repositioning/Redeployment of Old Equipment.............................................................210 Secondary Market for Old Equipment.............................................................................210 Donation of Old Equipment.............................................................................................211 Revisiting TCO and ROI Models for your organization .....................................211 Security, Privacy, and Safety .......................................................................................................212 Emergency Calls—911, 999, 112, and So On .................................................................213 Telephony Migration Checklist ...................................................................................................213 Summary ......................................................................................................................................213 Telephony Migration Checklist ...................................................................................................214

x

Copyright Statement

Copyright Statement © 2008 Realtimepublishers.com, Inc. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtimepublishers.com, Inc. (the “Materials”) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtimepublishers.com, Inc or its web site sponsors. In no event shall Realtimepublishers.com, Inc. or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, noncommercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtimepublishers.com and the Realtimepublishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtimepublishers.com, please contact us via e-mail at [email protected].

xi

Chapter 1

Chapter 1: Forward…Into the Past! Telephony is changing. Packet-based voice is inevitable. Being in traditional telephony today is akin to being a telegraph engineer in the late 1880s—wondering what that odd couple Bell and Watson were doing with your telegraph cables late at night. Only in our case, there are no thick metallic cables strung from wooden poles. Today, we have wireless signals completing the trip that started on thin strands of glass and may end on MOCA-flavored coaxial cables or twisted pairs of copper wire in residences or smaller fibers in the walls of offices, dorms, manufacturing plants, and laboratories. You are reading this guide because you already know that the traditional telephony era is drawing to a close and you are looking for guidance on how to migrate to the next era as quickly and painlessly as possible—maybe personally as a career move, or professionally on behalf of an enterprise or other organization. Chapters 2 through 8 of this guide are all about how to do just that. But first we must set the stage. Speaking of setting the stage: several years ago, I saw a Broadway musical called Pippin, about the son of Charlemagne. Recalling a Broadway musical about the life of the son of Charlemagne must seem an odd way of starting a book on the successful deployment of Voice over IP (VoIP) and IP Telephony (IPT), but stick with me, there are some interesting parallels. For one, the story of Pippin is one that is fairly well known, albeit with its share of misconceptions and historical fuzziness, similar to the 100+ year history of telephony. The musical tells the story from a unique, yet enlightened, and enlightening, point of view: it chronicles Pippin’s personal search for excellence and his desire to make an everlasting contribution. Lastly, at the very beginning of the musical, the Lead Player requests, “I beg you; cast all previous misconceptions aside and accept what we enact for you today.” In this same spirit of embracing a new way of looking at things you think you already know, cast aside your previous misconceptions as you read this guide. This book represents a fresh, and I hope enlightened and enlightening, view of the next evolutionary step in human communications. This guide explores beyond simply replacing dial tone on circuits with dial tone in packets, though it addresses this basic function because basic dial tone is often all that an organization needs. This book challenges a lot of what the market has been feeding us and a lot of what we think we know and “remember.” The view is a simplistic one, really, which asks questions such as “Why are we doing this?” and “What is our Return on Investment?” This book represents a view based upon experience, based upon listening at least as much as talking, and based upon thinking—probably way too much thinking—about why things are being done the way they are and less about starry-eyed amazement at what technology can do and how it will transform society and forever change the way humans interact with each other. This book is written neither for “Bell-head” nor “packet-head” but rather for the next generation of communications professional who will recognize these former terms only in their historical context, “back when” there was a difference. Today’s communications professional is more interested in session initiation, and has a protocol for such things; they are interested in voice as one of a rich palette of communications methods to solve a specific set of problems in the bigger picture of instantaneous access to information generically called multi-media, or often called the voice, data, and video triple play. Oh, and this is the first and last time I will mention H.323 because after this chapter we will only be looking ahead, not backwards.

1

Chapter 1 To further set the scene, please allow me to enumerate the observations that form the foundation for this book. The basic underlying premises: •

In the early phases of VoIP, there has been a tendency to accept mediocrity for cost savings and to embrace lower quality for convenience, but this will soon give way to quality as a differentiator of products and services.

•

“Free” is a great concept but is not sustainable, and VoIP is not often as cost-effective, or as nearly “free,” as advertised.

•

VoIP is a stop along the journey to packet-based telephony, not the destination.

•

IPT is the real objective for business and organizational users and will revitalize, rather than kill, all the best aspects of traditional telephony.

•

Telephony is becoming an application like spread sheets, word processing, browsers, and Instant Messaging, and, like the other applications, has common elements as well as special requirements needed to make it truly successful.

•

End-user organizations are shifting to some degree to playing the role of telecommunications carrier for their end users or, at the very least, providing a stronger, less transparent, liaison role between their user community and managed service providers.

•

Internal customers must be handled with the best practices that world-class carriers and service providers over the years have applied to their external customers, including Service Level Agreements (SLAs) and the underlying measurement and reporting of performance and service troubleshooting.

•

Traditional telephony knowledge and skills are as valuable now as they have ever been and are increasing in value as the number of people who possess this special knowledge is decreasing in the job market.

•

You might as well plan your trip to packet-based telephony and make the migration as smooth and painless as possible because, voluntary or not, you are going there anyway.

•

There is no need for most organizations to rush to packet telephony. Older telephony systems should be kept around as long as they make business sense and as long as they can be coaxed into providing dial tone.

This chapter begins with a high-level overview of electronic human voice communications; subsequent chapters delve into the details of what is needed to make the future better than the past, how—specifically—to implement business voice communications, and how to measure performance and use it as a part of a feedback loop for service optimization. The emphasis will be on midsize to large, national to global enterprises, but smaller users, carriers, and service providers will find their share of readily accessible knowledge in these words, as well. After reading this chapter, you will understand the basis for the rest of the book and will be better able to apply what you read in the remaining chapters.

2

Chapter 1

VoIP vs. IPT The dramatic shifts and uncertainty in today’s telephony marketplace can be attributed to two distinct phases of telephony technology deployment, each occurring at key points in time, with a very important transitional period occurring between the two. The two phases are the “circuit phase” and the “packet phase,” with an intervening transitional period I will simply call VoIP. To make the point clear, you can also call the circuit phase the uni-media phase and the packet phase the multi-media phase. This is not to say that no one did multi-media with circuits, quite to the contrary; rather, the distinction between the two phases focuses on the capabilities inherent in a single connection. The uni-media phase, a term invented by this author solely as a vehicle for making this comparison, is characterized by a single circuit delivering a single type of information: a circuit or channel for voice; another, separate, circuit or channel for data; and another, separate, circuit or channel for video. The treatment of each circuit as a single information pathway, largely independent of each other in transmission and possibly synchronized at the end points is fundamentally different than the multi-media view of treating all connection types as variations on the packet theme with appropriate allowances being made along the way for the needed Quality of Service (QoS) for each. In the uni-media phase the separation between different types of media is more rigid and enforced at a much lower layer in the network.

The Uni-Media Phase The uni-media phase of telephony is defined as beginning March 10, 1876—the invention of the telephone—up until mid-1994 when engineer/hobbyists who would go on to form VoIP pioneer VocalTec made the first PC-to-PC voice call from Princeton, NJ to Israel and VoIP emerged. The only major change in telephony up until that time was the shift from the analog network to the digital network. Though VoIP represents a major change, the fundamental achievement was still a point-to-point call from one human to another. The uni-media phase of telephony is characterized by fast, reliable, guaranteed QoS circuit switching, which was originally analog and eventually digital. The first phase delivers a 99.99+ percent reliable range of services to most areas of the world through a vast, carefully constructed global network based on standards that segregate the circuits into their own, dedicated, protected uni-media lanes in the network highway. The uni-media phase is also characterized by a rigid, standardized, hierarchically structured geographically aligned numbering system designed in five layers that allows any-to-any connectivity globally with a minimum number of steps.

3

Chapter 1

The Multi-Media Phase The multi-media phase is disruptive and challenges the established order but goes beyond VoIP. VoIP was thought, at one point, to be the multi-media phase, but VoIP has turned out to be more of a “living laboratory” and has become a transitional step from uni-media to multi-media and not a full-blown revolution unto itself. More than anything else, the advent of VoIP kicked off the new thinking about the future viability of circuit-switched telephony in light of customer demands for integrated multi-media applications, such as presence and unified messaging, and put us on our current trajectory toward all-packet/non-circuit multi-media networks. The multi-media phase cannot rightfully be called the multi-media phase “of Telephony” because at least two other media are involved, each having a variety of flavors. Beyond voice, there is data and video, and each has real-time, near-real-time, and non-real-time variants. The multi-media phase is represented by the ad hoc use of the global Internet and other IP intranet and extranet networks for a mixture of telephony, data, and video in a service package often referred to as the “triple play.” This direction is as true of residential or personal communications as it is of business communications, the difference being that in the residential context, the triple play has a decidedly entertainment spin, and in the business realm, the emphasis is on strategic and tactical applications that will either save or make money. At the present time, early in the multi-media phase, Internet telephony services are immature, network reliability is lower than in the uni-media phase, and QoS—and the resulting Quality of Experience (QoE), are measured differently than in the uni-media phase, yet there are some compelling reasons to move to a packet-based architecture. Among those reasons are true ondemand multi-media applications; transparent service ubiquity over metallic, wireless, and fiber delivery systems; and lower cost/higher efficiency consolidated (I dare not say converged) backbones, all of which show a lot more business benefit and potential longevity than the early VoIP cost-focused discounting models that have already failed. As we move further into the multi-media phase, next-generation carriers will start to compete on services, not price, over both PSTN and IP network infrastructures. This is the hallmark of the next phase—a value-added services framework that delivers enhanced telecommunications capabilities to the marketplace as services, not as boxes and wires that must be further integrated by customers to provide services. And with a scalable and flexible enhanced service architecture solution riding atop these new-generation networks, it will actually be possible for the IP-centric network implementations to increase their competitive advantage over circuit-switched network providers and beat the analyst projections for market share and revenue growth.

4

Chapter 1

VoIP Traditional telephony had changed little in concept or common practice from the very first telephone call in 1876 until the first VoIP call in 1994. The major breakthroughs had been the introduction of digital switches, implementation of digital telephony, use of fiber optics for communications, and the use of out-of-band signaling methods, such as SS-7/CCS-7 and the ISDN D channel. These are all evolutionary improvements on the basic theme of an end-to-end circuit switched call. Other important recent changes have occurred in the regulatory environment. These implementations were made possible by advances such as fiber optic networking and SS-7 signaling. Were it not for the advent of the Internet and IP-based telephony efforts, traditional telephony would probably have stayed the course for many more decades to come. However, traditional telephony and IP networking did cross paths, which, concurrent with deregulation in the United States and elsewhere, caused a number of tangible changes. The largest change has been the precipitous decline in the cost of circuit-switched long distance service, in not only the United States but also in many other countries around the globe since 1995 to present. Where 20-to-25-cent/minute prices for long distance calls were common in the U.S. in 1995, now sub-2-cent/minute prices are the norm with companies such as major disruptor Vonage offering residential flat-rate plans to the U.S., Canada, and Puerto Rico for US$24.99 per month and similar business plans for $49.99.

Technical Definition vs. Market Definition A careful analysis of the foregoing discussion speaks to the issue of the two distinct definitions for VoIP. One definition, a market definition, includes a general, collective, label of VoIP for all the market, legal, and regulatory forces and technologies that have caused the revolutionary and dramatic changes in global phone calling, freeing the consumer or business user from having to understand the details. The second definition, the technical definition, includes a vast number of technological innovations that have enabled the business definition but of which the thing called VoIP is only small part. There is not, for instance, a VoIP protocol, per se, but rather a number of choices, all of which will be examined, contrasted, and compared in subsequent chapters.

5

Chapter 1

VoIP is Revolutionary VoIP began in what can best be described as the hobbyist realm. Early VoIP software was produced in spare bedrooms, garages, and office cubicles and used existing hardware that was cobbled together to accomplish the task of putting voice over the ubiquitous and inexpensive Internet without any regard for, or in most cases knowledge of, traditional telephony switching, signaling, or other systems considered vital to high-quality, highly reliable universal telephony. Use of the primarily flat-rate Internet, instead of the metered telephone network, allowed VoIP users to avoid paying long-established tariffs on the traditional local, national, and global voice networks. The desire was to put voice on the Internet as one more application of the globally connected Internet, which could lower costs and, potentially, put more people in touch with each other at a lower cost than keyboard actuated IP tools such as email alone had been able to do. Calls in this phase were mainly PC-to-PC. The second step of the transition was “toll bypass,” and was predominantly the use of existing telephone hardware with customized software, which allowed avoidance of toll charges and a growing perception that VoIP was basically regular toll voice at a lower price. Calls in this phase continued from PC-to-PC, phone-to-PC, and PC-to-phone, but began to include phone-to-phone calls using the Internet for the long-haul, long distance part of the call. International calls were offered at particularly attractive pricing. Toll bypass naturally morphed into “free voice” for everyone. The result of the widespread news of “free voice” was that traditional carriers slashed their prices to retain customers and avoid disappearing completely, which turned out to be an over-reaction on their parts, but appeared to be an entirely sane move at the time. In retrospect, we see that price reductions were mandatory, but a less instantaneous, more gradual approach would have still worked in the marketplace and would have presented less risk to the financial health of the traditional purveyors of voice services. In turn, lower prices for traditional telephony had a significant dampening effect on new world carriers planning to implement VoIP networks because most of the initial economic models of early VoIP implementers were predicated on providing a lower quality and reliability service, compared with traditional circuit-switched phone networks, but at significant discounts. These plans were put into question when the traditional, high-quality and reliability services became available but at competitive prices. The biggest problem with VoIP is that residential and business customers alike are learning through difficult, real-world experience that VoIP is not “what you were getting from the phone company, just cheaper.” VoIP lacks the reliability, scalability, standards, signaling, services, ubiquity, and capability of the uni-media phase. What VoIP pundits fail to realize is that it is not possible to reproduce in a weekend what it took over a century to produce in the uni-media phase and then to easily apply it to multi-media applications, many of which are not yet fully understood. VoIP, though, does provide a real-world telephony laboratory and a natural bridge to the multi-media phase, where the real issues of transitioning to a new world order of Telecom are addressed.

6

Chapter 1 It turns out, however, that the keys to success in the coveted multi-media phase lie not as much in the hardware but in the higher-layer application software architecture, including the Session Initiation Protocol (SIP), IP Multi-Media Subsystem (IMS), and Parlay-compliant Applications Programming Interfaces (APIs)—and there is still a lot to be done before the current state of the art replicates what we currently have in the uni-media phase, let alone surpasses it and brings new and engaging applications to light. This guide will use VoIP in the more generic sense, but will focus on more specific items under the umbrella of IPT. Why? Because VoIP was originally intended for PC-to-PC voice messaging and was never intended to replace global telephony. The idea of “free voice” was compelling and simple, yet the model is not sustainable. There is still the underlying fact that someone has to pay for the global Internet to carry voice and that as a niche, hobbyist toy it is possible to sneak some voice in with data, but in the quantities that are needed to fully shift global telephony traffic to the Internet, the voice packets will be noticed and must pay their own way. VoIP is really more about the edge of the network. VoIP, in our definition, is a very visible, tangible thing and requires new devices and a change in the traditional ways of communicating. It requires PC-based agent software, SIP phones, or PDAs with VoIP capability, but is this not just a new way of providing dial tone? And, if so, and this author feels that it is, is there anything compelling about VoIP from a business standpoint? Does it enable you to do anything that you have not been able to do before? Not in its early stages, but it holds out the hope of advanced applications in the future and it is the new, advanced applications that are needed to make this thing generically labeled as “VoIP” to be successful. Multi-media and wireless are crucial to VoIP success, as VoIP plays an important role in voiceenabling multi-media and the traditional data-only wireless systems already in widespread use in business and personal communications.

VoIP Business Aspects Not Clear VoIP may or may not cost less than traditional telephony, but that is not always the main objective. Take the case of Heritage Bank of Alabama. Heritage was faced with a choice of not being able to adequately support their growing client base or investing in old PBX technology or taking a risky step as an early adopter of VoIP. Although VoIP systems have dropped in price and the quality and variety of sources of implementation expertise have grown since Heritage first took the leap, it is still true that the changes in training, operations, culture, and user acceptance all represent big hurdles. It is also true, as shown in the following case study, that a shift to VoIP is often inopportune and does not always yield the compelling 30 to 40 percent benefit demanded by today’s businesses. In many cases, the issues are more practical than a compelling savings and are more about the survival of the business because voice communications is a fundamental service without which a business cannot function. Shown in this light, a savings of less than $10,000 per year for a regional bank is better than a loss but is not the focus.

7

Chapter 1

Growing Up with VoIP at Heritage Bank Heritage Bank, based in Birmingham, Alabama, realized some cruel facts about being forced into VoIP prematurely by exhausted traditional technology. There are more lessons than may be obvious for any company that is not considering the age and obsolescence of the existing system. ROI The manufacturer gave Heritage Bank price concessions in compensation for substantial problems early in their implementation cycle. The ROI presented takes into account the price concessions but not price reductions since 2002 and, therefore, may represent a reasonable expectation of ROI for a system that is implemented in today’s competitive pricing environment. The following rough-order-of-magnitude ROI analysis shows the hard savings the bank has realized from their consolidation of service providers and shift to VoIP. In addition to the hard dollar savings, there are increases in customer satisfaction, system supportability, and security and enhanced internal user confidence that are impossible to measure. Heritage Bank’s ROI has two components, both of which complement the other: service consolidation and VoIP. It is customary and prudent to separate the ROI components to clearly show the impact of the service consolidation and VoIP separately. It can be argued that VoIP can be implemented separately from a service consolidation and vice versa. In most cases, this is true, but in the case of Heritage Bank, there were so many inter-related elements that the project must be taken as a whole. A further argument could be made that the bank could utilize VoIP-enabled low-cost voice services to further lower their costs. This may well be a future initiative but the bank is extremely happy with the quality of their current service and support presently being provided and will approach such an initiative very warily. Such an option was unrealistic in 2002. A 46-month ROI is high; the norm today being 24 months or less. What the ROI hides, however, is that the old system was at its limits: it was unable to sustain Heritage’s growth. An outside observer could quickly tell the strategic importance of this project to the bank by the fact that they spent just more than US$200,000 for consulting and services. TCO A TCO comparison of the old key system and the new VoIP system provides further evidence that the bank’s move was more strategic than simply cost savings and correlates well with the ROI analysis. Total savings after 5 years, not adjusted for inflation or any other economic factors, was a mere $40,000, but the bank is still in business and thriving.

8

Chapter 1

IPT IPT was born of VoIP and proved compelling for traditional carriers, Internet Service Providers (ISPs), and end-user organizations with stricter TCO and ROI guidelines for new technology adoption. Right out of the gate, IPT was intended for both PC and non-PC device interconnection, was designed to replace traditional telephony because it incorporated key functions ignored by VoIP—such as signaling, and can be used as a transitional technology harmoniously and seamlessly providing a telephony environment of mixed legacy and VoIP devices. IPT is complex but well understood and can be successful simply by providing traditional dialtone over cost-reduced multi-media packet infrastructure. IPT can be used as a stepping stone to a future multi-media and wireless environment, or can be used to create a mixed environment. IPT can provide cost reductions of 20 to 40 percent over traditional telephony, and TCO and ROI models are well understood, as shown in the two examples that follow—one is an example of a packet voice infrastructure at a county school system and the other is a PC-based application at a global consulting firm. IPT at Paulding County Schools: A Formula for Success Paulding County School District in Paulding County Georgia has implemented a school system-wide VoIP setup with an IPT emphasis incorporating almost 600 phones and completely replacing their older PBX and CENTREX service and the entire physical infrastructure in all schools. ROI There are three ways to approach the ROI for this project. The first model is Voice Rides Free, the second is Port Allocation, and the third is Capacity Allocation. Sources of Savings: Although the system provides additional “soft” benefits, the ROI and TCO calculations are treated as if this is a straight functional replacement of a phone system. No savings will be calculated for the move of PRIs to USLEC—this could be achieved without a migration to VoIP. Voice Rides Free: The first model assumes that the system needed to be installed for data and, because it was so substantially overbuilt, the VoIP traffic represents a small impact. Therefore, the ROI can be calculated as the cost for VoIP systems and phones of $450,000 divided by the $33,000 per month savings to yield an ROI of 13.6 months. Port Allocation: The alternative to Voice Rides Free is for both voice and data to share the costs of the system. In this first allocation model, Port Allocation, costs are allocated based on the total number of ports. If you were allocating the costs of a road system, this would be the equivalent of splitting the total cost by the number of driveways, regardless of the type of vehicle that would use the road. With a total allocated cost of $716,850 and a savings of $33,000 per month, the ROI would be 21.7 months. Using these figures, an organization considering a new system can calculate the rough costs of their own converged voice and data solution. Keep in mind that financial modeling shows these costs to be roughly consistent with the costs of a traditional PBX, so these figures are not valid for comparing VoIP and PBX. If a change of telephony system is mandated, the numbers show a move to VoIP is viable, while a more reasonable decision point might be the answer to the question “Are there compelling reasons for a move to a new telephony system now or can we wait?” Capacity Allocation: Capacity-based allocation takes into account the demands placed on the network by voice and data and shares the cost of both network use and network idle capacity. If the network were a road system, this is the equivalent of allocating costs based upon the size of the vehicles using the road. Based upon an average demand of 384 thousand kilobits per second (kbps) per data user and average demand of 70kbps per voice user, we could allocate 18 percent (70/384ths) of the infrastructure costs (18 percent of $1,452,000 or $261,360) to voice, add the $450,000 cost of VoIP equipment, and divide the total of $711,360 by the $33,000 of savings per month to yield an ROI of 21.55 months. 9

Chapter 1 TCO TCO is always a tricky calculation but a very useful one, especially when used to compare two or more systems for which the TCO has been calculated using the same algorithm. In the case of the Paulding County School District VoIP project, the system life of 5 years will be used for calculation of all recurring costs. Because there is no dedicated staff, no personnel overhead will be calculated, but the outside maintenance fees will be considered. A useful figure for comparison purposes is the total cost/user, and this will be calculated based upon 450 users. Fixed costs will be based upon the Port Allocation method shown for the ROI calculation. Based upon these rough order of magnitude calculations, for budgetary purposes, the cost to provide VoIP-based phone service for a single Paulding County School District telephony user is $2953 for 5 years, or $49 per month; a figure substantially below the $74 per month cost of the Centrex service being replaced. There are two forces at play. The first is the usual impact of replacing a 10-plus year-old technology with a new technology: the double impact of a lower price and more capability. The second force is the financial impact of Paulding County’s very favorable monthly fiber cost. With normal fiber pricing, a budgetary figure of $80 to $100 per month per user to provide VoIP-based phone service would be more realistic for an organization of similar size, which would mean that even traditional Centrex may be more cost effective and the cost difference may have to be justified based upon strategic application benefits or on a “Voice Rides Free” strategy, which also makes sense for many organizations. This example clearly highlights the importance of keeping infrastructure costs under control.

The Paulding County Schools example is a compelling one, especially when considering the TCO on a per-user basis. Calculating TCO per user allows a direct comparison of costs from the IPT system to the traditional system and provides very convincing evidence that Paulding County Schools made the right decisions along the way. This example shows that much of the value of IPT projects can come from the infrastructure part of the project and that all aspects of the system implementation are inextricably interwoven and must be thought of differently than traditional voice or data or video projects—projects often accomplished by multiple standalone departments. The next financial case study represents substantial savings with a shift to PC-based IPT. A group of globally dispersed consultants were able to use VoIP application software to overcome the inherent limitations of their cell phones, leveraging the global ubiquity of 802.11 LAN standards. One can only imagine the positive shifts that will occur as WiFi hot spots continue to proliferate globally, and how the convenience and ROI will improve as the engineers increase their use of wireless for VoIP, data, and video.

10

Chapter 1

Semco Maritime Does Global VoIP—Without Phones Semco Maritime, a global engineering services firm primarily serving the energy and shipbuilding industries, has harnessed the power of IP to create a high-quality, money-saving, PC-based VoIP service for their 600+ geographically dispersed employees. The two-tiered service combines MPLS and Internetbased VPNs in a true VoIP solution. ROI ROI for Semco Maritime was only a secondary consideration: they required global voice connectivity but soon learned that it was accompanied by an impressive ROI: the initial investment of less than 600K Euros was paid back in 4.8 months. This ROI is based on the clients’ own numbers and savings figures of 200 Euros per month per engineer. In the author’s anecdotal experience, hotel room phone charges are much higher than the 10 or so Euros per night saved on an average 20-hotel nights per month. My own calculation is that at least 30 Euros per night would be saved, except by the most frugal and disciplined engineer, who always makes calls from a lobby phone with a phone card and does not use the room phone. If this were the case, the ROI would be 1.6 months.

TCO Although the TCO calculation shown below could easily be challenged by any competent accountant, it does provide a rough order of magnitude idea of Semco Maritime’s TCO for this application. Additional considerations could include allocating current IP data costs to voice based upon percentage of traffic or bandwidth and similar calculations, but those arguments could be countered by saying that the data expense already exists and, therefore, voice rides for free. The fact remains that Semco Maritime’s total cost per user per month for as much global telephony as they can use is in the range of 20 to 30 Euros.

11

Chapter 1 An analysis of companies moving to IPT, in most cases, shows a decided price/performance advantage. Price/performance advantage tends to highlight the word “price,” or more importantly for carriers, service providers, and larger end-user organizations providing services to internal customers, the underlying “cost.” The most important aspect of the multi-media phase, however, must be the price (or cost) to performance ratio. It seems to go, almost without saying, that the economies of scale and benefits of integrated network measurement, troubleshooting, management, and operations created by running voice and other enhanced multi-media services over high-bandwidth shared IP networks should be able to provide efficiency at low cost. What we have learned during the transitional VoIP period, however, is that it often does go without saying, but shouldn’t, as the “converged network is more efficient and lower cost” argument does not always work: problems and unforeseen costs scale up as well as savings in large networks and are magnified for larger organizations—and not always in direct proportion to the size of the network. For many organizations who have maintained a largely unchanged traditional telephony infrastructure, there is an immediate and positive impact of a change to a new, modernized system simply due to Moore’s Law, which states that there is a natural price reduction accompanied by an increase in capabilities that is simply a byproduct of the advancement of technology. Beyond that, wild claims of “free voice” and “elimination of voice personnel” have created unrealistic expectations that are very difficult to overcome, often leaving those who are to finish crossing the VoIP bridge to the multi-media era with insurmountable problems, primarily in terms of user and management expectations that are critical to a successful transition. Without proper expectation management, any project, and especially one that impacts such an important capability as human-to-human communication, is doomed to failure. This includes proper expectation management in terms of cost savings, operational simplicity, voice quality, and many other areas.

Major Changes Sorting out all the changes and putting them into categories, we see that the two major changes are that telephony is becoming an application, rather than a service and that end-user organizations are accepting more responsibility for voice applications than even organizations with historic PBX infrastructures. We can also observe that the shift from a service, provided largely by outside organizations, to an application, traditionally managed by internal organizations, also heralds the second major change, a shift in responsibility placing the larger end-user organization in the role of carrier or service provider to internal clients.

12

Chapter 1

Telephony Becomes an Application It has long been assumed by the IP crowd that the migration to Everything over IP (EoIP) is almost an evolutionary “given,” more so, even than a revolutionary coup, but the arguments have been more emotional than real. The argument has always pivoted around the superiority of IP over circuits coupled with the name recognition and market acceptance of the Internet and its underlying protocols. The piece that has been missing, however, is that the real benefits of the Internet, and the market’s fascination with it, come from its simplistic interfaces and software capabilities, and not from its underlying architecture. If the truth be told, wouldn’t each and every user want a dedicated, guaranteed-performance path from themselves to each other system with which they interact? And isn’t this exactly what a circuit offers them? No, users are not enamored with IP and its switching characteristics; they are in love with the applications, which can be characterized by two words: simple and multi-media. And these applications will be offered in packages that are simple and easy, possibly under the banner of Virtual Private Network (VPN) services, and quite possibly offered seamlessly over a variety of transports, both wireless and wireline. Migration from traditional telephony to the multi-media Voice over Packet model should be strictly an internal network issue and should be transparent to the actual user of the service. The migration should be seamless, and the only evidence that a subscriber should have that the migration has occurred is that they now have a menu of new, meaningful multi-media services that were not previously available. Another key point of consideration as end-user organizations take a greater responsibility for their voice services is that monitoring and management tools are no longer embedded in the network but rather form a new part of the skill set the end-user organization must adopt. As the responsibility for services shifts from the carrier and service provider to the enterprise organization and eventually to the end user, and as the end user becomes empowered and more sophisticated, so too, must the monitoring and management tools provided by the network. If there is one thing that VoIP has taught us, and a mistake that cannot be repeated, is that although it is possible to render voice waves and bits and to put them into packets that there is a lot more to the entire puzzle than just this. Voice is an IP application, but it is not “just another application.” We have learned that “cheap” alone won’t make voice on the net work, and that the next phase must include much more of traditional telephony monitoring and management, coupled with new tools, than we ever thought possible before.

13

Chapter 1

Responsibility Shifts from Carrier to End-User Organizations The technical challenges of running advanced enhanced services across large, often multi-carrier, IP networks are magnitudes of order higher than providing vanilla VoIP PSTN-replacement call connectivity. When implemented by “traditional” enhanced service platform providers, the database-centric execution of many popular enhanced services, such as phone/debit calling cards and ubiquitous unified messaging, holds the unpleasant promise of introducing new orders of call latency that equipment suppliers just thought they were close to “solving.” Within the VoIP transition, the actual users have only had a glimpse of the problems, while the service providers face scalability issues every day. The first-generation client/server service platforms have extreme scalability limitations, typically being able to run a half dozen or so “remote” switches before they experience melt down. There are, at present, no VoIP or multimedia phase networks that even approach the scope or scale of traditional uni-phase networks. Consider that the leading VoIP provider, Vonage, just passed a million subscribers in Spring 2006, and that a million is not equal to a single midsize United States city. In addition, existing VoIP signaling protocols don’t address potentially service crippling latency issues within VoIP networks. To more effectively meet latency requirements, multi-media networks must not only provide fast database access but also distribute control intelligence throughout the network, a lesson learned from uni-media. We have also learned that multi-media networks are complex organisms requiring specialized monitoring intelligence, and that manufacturer-provided monitoring tools are a part of the solution but do not take into account the nuances and subtleties of the overall system. However, generic monitoring and management tools are insufficient—to be truly effective, third-party tools must be constructed with a knowledge of the individual manufacturer’s systems being maintained in order to provide a system-wide, end-to-end view of system performance. The key areas that next-generation monitoring and management tools must cover are: •

Pre-deployment simulation and modeling

•

Manufacturer-specific monitoring and management

•

Real-time business views

•

•

Call detail records

•

Calls in progress

•

Delay-to-dial-tone rates

•

Gateway channel utilization and loading

Real-time call monitoring •

Phone and multi-media device availability and monitoring

•

Successful vs. failed call completion rates

•

Poorly performing components

•

Service level breaches and SLA compliance

•

Real-time interface to Manager of Managers (such as HP OpenView)

14

Chapter 1 •

Summary and exception reporting •

Utilization trends over time

•

Managed devices by company, department, and location

•

Asset tracking

•

Capacity planning •

Incoming and outgoing calls

•

Loading by dial plan, routing rules, and gateway

•

Bandwidth utilization

•

Delay and delay variation

•

Packet loss

•

Route patterns, utilization, and availability

The IPT Life Cycle In order to be successful in your next-generation network venture, expectations should be set as follows. Only in so doing will the results you achieve have any hope of being judged a success when compared with the expectations you set: •

Users—Different is not bad. The new system will initially provide all the important functions of the current system and then, after the baseline is established and the migration is complete, will provide important new capabilities. The system will sound different, but this is not necessarily a bad thing. The system voice quality will be (not as good as, nearly the same as, better than), fill in the blank based on your objectives, and the new system will be more flexible and give you more freedom while allowing you to stay connected.

•

Management—The cost of the new network, when taken as a whole, may be the same or higher than what we are paying now but we must move forward to assure the continuity of our voice communications and eventually acquire advanced capabilities of strategic benefit, though our unique combination of geographic coverage and scale may allow us to realize overall raw cost savings. There will be four specific phases, each with its own specific objectives. Investing more time, effort, and attention in earlier phases will lower the overall costs and increase the value to our organization. The phases are planning and assessment, pre-deployment testing and implementation, ongoing operations, and optimization. Contrary to popular belief, VoIP is not “free voice” and VoIP is not “just another application.”

We will now take a closer look at each of these phases in the IPT life cycle.

15

Chapter 1

Planning and Assessment The objectives of the planning and assessment phase are to develop a replacement system that initially delivers replacement functionality and eventually adds important strategic functions. The shift to the multi-media phase is characterized by a new, fresh commitment to traditional telephony characteristics, such as reliability, security, and fraud control and customized services deployment. In addition, desirable new aspects—such as a multi-media focus—will be available on a large scale. All of this must be taken into account in the planning and assessment phase. It can also not be stressed enough that management expectations must be set so that a sufficient amount of time and resources can be invested in this phase to make the phases that follow it successful. VoIP is not just another application that can be launched by plugging in a few IP phones. A proper planning and assessment phase will take all of the needed factors into account.

Figure 1.1: The path from uni-media to multi-media.

“Making the move” to multi-media is not a very accurate statement as those carrier, service provider, and end-user organizations adopting multi-media will not, in most cases, actually ‘move’ at all. In most cases, they will already be providing or using traditional and/or VoIPbased systems and will simply transition to multi-media non-disruptively. The first step is to ascertain the current services and features that are being utilized in the unimedia phase to assure their accurate reproduction in the multi-media phase. A methodology for accomplishing this step is presented in the next chapter. The next step is to move as quickly as possible through the transition phase, though many organizations would be well advised at this point to skip over the VoIP step and move directly to the ultimate converged IP-centric multimedia IPT phase. This step avoids the risk of being in the transition period and reduces negative user impact. 16

Chapter 1 Pre-Deployment Testing and Implementation The objectives of pre-deployment testing and implementation are to assure the outcomes from the planning and assessment phase and to determine the impact of adding the new applications to the existing network, then to roll out the new system into the organization. The use of sophisticated, specialized modeling and planning tools in the first two phases will allow us to assure the maximum benefits in the shortest amount of time with the least possible disruption to our operations. Ongoing Operations and Optimization Ongoing operations and optimization are two phases with opposite objectives. It is the objective of the ongoing operations phase to ensure consistent functionality across the network, regardless of where an individual is in the organization geographically. Ongoing operations is constantly measuring the network’s performance against established benchmarks to ensure unwavering compliance and absolute consistency of network services. Optimization, in contrast, has as its primary objective improving operational aspects of the network services and adjusting the operational baselines to reflect new baselines. One of the key items that will require revisiting, and refreshing, is the SLA. The SLA is an effective tool the full potential of which has never been realized. In its most basic form, the SLA is a contractual vehicle that, at the very minimum, documents the minimum level of service that a customer is willing to accept before they are due some penalty or refund on their service, but it can, and should, be so much more. It can, and should, also provide a description of the actions taken by a carrier or service provider should the user not keep up their part of the arrangements, such as using a service class for which they have not contracted. SLAs should also be automatic. At the present time, too few carriers and service providers have embraced the SLA as a strategic customer relationship management tool and require the customer to jump through hoops to get the penalties they are due. SLAs and the compliance information supporting them should be available to the customer when and as needed. We will see a lot more of this in the multi-media future, and sophisticated monitoring and management tools will be required to keep customers apprised as to the QoE delivered by a network, versus what was promised and contracted. In fact, the education process that goes along with this effort should be initiated as early in the process as possible because the process of educating the user on SLA compliance is also the process of educating the user on how to select multi-media services without relying strictly on price. The process about which we are speaking gets to the heart of the matter of creating a more informed consumer of multi-media services in such a way as knowledge about vehicle performance makes a more informed purchaser of cars. Mean Opinion Score, PQSM, R-Value, and similar voice quality metrics should be as familiar to the consumer of voice services as MPG, horsepower, and fuel options are to owners of vehicles. And this analogy makes additional sense not just in terms of SLA performance but, as previously mentioned, in terms of a more informed purchaser who will forego the commodity approach of “voice by the minute” or “gas by the gallon” for a more informed purchase decision based upon metrics other than pure price. Tools that allow the monitoring of voice traffic and related quality metrics in real-time and provide feedback to suppliers and consumers on an exception basis, using baseline thresholds established in the SLA, are crucial to success in the multi-media future, as they keep the focus off price as a singular differentiator. 17

Chapter 1 One possible model that might emerge is that transport becomes a commodity over which differentiated services flow. In this model, the prior performance of various transports, representing some combination of different QoE offerings from one or more transport providers, is known and the closest QoE to that needed by a given service at a moment in time can be selected, possibly on a “pay as you go basis.” An alternative to that would be that bids would be requested, in real time, from available transport services to provide a certain QoE at a moment in time. How would this work in practice? Suppose that you were in a coffee shop in Milan and needed to make a good night call home that required high-quality, bi-directional audio and real-time video. You would have already made whatever requirements were needed with the coffee shop—free wireless hot spot, pay-as-you-go, used your corporate account with the wireless provider, whatever was needed—for access to the Internet. In this case, you would select the called-to party on your PC/handheld/wireless video phone and a request would go out for the long-haul service needed to handle your call. Because of the low-loss, high-quality needed, you would basically be negotiating for a toll road for your call. The negotiation would be made with a transport service that would provide what you needed, at a negotiated price of .7 cents per minute, right now with no guarantee of this price in the future. Your measurements from prior calls, and/or other users, would be used to validate the bidder and your call is put through. The next call might be a highly compressed voice-only call to check voicemail and might not require the toll road. There might be no additional cost for this service beyond what is included in your basic monthly rate. Anything is possible in the multimedia future. Optimization While the ongoing operation phase is about consistency, the optimization phase is about a regular, constant, ongoing program of positive change in network operations. Optimization takes the measurement and SLA compliance statistics gathered during ongoing operations and seeks ways to improve those statistics with specific target outcomes. Desirable objectives will include financial or operational goals, such as releasing resources on traditional telephony networks as calls shift to the VoIP network or even reducing the amount of bandwidth used per call on the VoIP network as newer voice coding techniques are implemented, both of which can have favorable operational and financial outcomes. The process will involve conceptualizing improvements, validating assumptions using modeling or simulation tools, then testing them in a laboratory or isolated network segment before implementing the changes in the actual network. After network performance has been assessed and any needed fine tuning done, the network operational benchmarks are adjusted and new optimization goals are established.

18

Chapter 1

Summary We are at the end of the first chapter which, as a standalone white paper, would simply be an overview of the possible future of telephony: interesting and academic and about as useful, or useless, as dozens of other white papers on the topic. But, this chapter is so much more. This first chapter has documented the underlying beliefs, philosophies, and observations needed to understand the rest of this guide and will be the last we will see of high-level concepts, abstract ideas, and sweeping generalities. From this point forward, this guide will morph into more of a practical guide, a “how to guide,” based on knowledge gained from planning and implementing VoIP and multimedia communications for some of the largest multi-national and global organizations. Chapter 2 will cover the IPT life cycle, and the following six chapters will follow the life cycle as we describe the processes, best practices, and pitfalls at each step, including planning and assessment, design and pre-deployment testing, implementation and migration, ongoing operations, optimization, and some very nitty-gritty details needed to assure success. Thus, if you are new to telephony or are an industry veteran, stand by for what we intend to be the most detailed, in-depth, and useful guide to VoIP and IPT implementation that it is possible to cram into 200 pages.

19

Chapter 2

Chapter 2: The IP Telephony Life Cycle Every system has a rhythm, a natural life cycle. If you can understand a system’s rhythm and “go with the flow,” your job of managing that system will be made much easier. The traditional telephony systems that we are now replacing have a natural rhythm, as do the IP Telephony (IPT) systems with which they are being replaced. This chapter is about understanding the life cycle of your enterprise IPT system and harnessing that understanding to increase the success of your implementation as well as your financial rewards—be those rewards direct tactical benefits such as cost savings or indirect, and often elusive, strategic benefits. This chapter is also about squeezing as much use as you possibly can from your existing telephony systems, a move that can make a lot of business sense regardless of your natural desire to discard the old and to start using the shiny new thing.

The Bigger Business Picture From the 1960s to the early to mid 1990s, a technology project often took 12 to 18 months or more with little variation in schedule from large to small organizations. Any technology project was a major undertaking and included an exhaustive process of a Request for Proposals, presentations, site visits, and analysis, often by an internal staff and a cadre of outside experts. In many cases today, however, it is only the “Fortunate 500” who still have sufficient resources to be able to take a careful, project-oriented approach and to think and rethink the various outcomes of possible choices or approaches—and often even those organizations will not apply sufficient resources to assure success. These days, smaller organizations are often forced to react rather than plan and to put out fires rather than building a fire-proof system in the first place. The following section explores the facts surrounding a project that was carefully thought through, argued out, and managed in a thorough and professional manner as an example. The project described was the development of a global Voice over Packet (VoP) blueprint and subsequent implementation for a Fortune 100 client. Although the client company is a global energy giant operating in 162 countries, there are lessons in this story that can benefit organizations of any size, industry, or geographic reach. After being briefed by the client on the intended scope of the project—a global migration to VoIP—my first step was to define terms and to redraw the scope of the project through a collaborative, interactive, and often heated and animated debate. The initial thinking on the client’s part was to create a new world of telephone communication that would connect all corners of their vast global organization relying largely on their existing IP network coupled with a new application called VoIP. Their primary motivation was to retire their old telephony gear and in fact, all their traditional telephony headcount and associated costs. Their drivers were as much a desire to get the benefits of “free voice” as anything else as well as to lower costs while maintaining the same level of voice quality and connectivity.

20

Chapter 2 A key part of the success of the project redefinition effort was the rewriting of the project objective. The objective was originally “to implement a VoIP system and reduce or eliminate costs for internal voice communication” and was rewritten as “provide a global migration path toward a multi-media network that will leverage existing technologies and systems, provide a baseline of voice, data, and video services and allow implementation of important new technologies when and where needed, leveraging existing assets, and to positively contribute to organizational success.” Even though the updated objectives sound a lot like a really bad PhD thesis title, the new objectives incorporated a variety of factors that must be included for the long-term success of any project: a consideration of how the project contributes to the long-range success of the organization and best utilizes existing assets. This project occurred in late 2003, and one interesting lesson learned during this early phase was how much things have changed since the 1990s. For instance, an effort to identify organizationspecific strategic and tactical benefits in order to establish milestones and success criteria for the move to a VoP system did not get very far. The Global Network Manager explained that in their world, tactical benefits were called “hard benefits” and hard benefits could be measured in dollars and cents. He further explained that strategic benefits were called “soft benefits,” and that there was no tangible way to measure strategic benefits. The project must, therefore, be measured strictly in terms of hard, tactical benefits. As a further historical footnote, he added that strategic benefits had not been considered since the late 1990s. Packet Telephony Is Inevitable Packet-based telephony is, in my strong view, inevitable, but VoIP, per se, is not. The first step was to redefine the project as the Voice over Packet (VoP) project, as opposed to a Voice over Internet Protocol (VoIP) project. The main difference is that VoIP originated as a way of allowing two people to communicate by voice between two computers while VoP includes a comprehensive toolkit that goes beyond VoIP to include other services required by traditional telephony, such as signaling, accounting, security, and enhanced 9-1-1 emergency calling. In addition, VoP extends the scope of communications to include a variety of services and capabilities envisioned in initiatives such as the Session Initiation Protocol (SIP) and the Internet Multi-Media Subsystem (IMS). VoP The broad term VoIP has been accepted by the industry, and consumers, to refer to a new, inexpensive way of making telephone calls using the Internet. Although “insiders” understand that the IP network is not always the Internet, the common definition is convenient and easy. What VoIP also means is that the voice information is encoded and sent, historically, using proprietary methods or the H.323 protocol—and, increasingly, via the Session Initiation Protocol (SIP). The term VoIP is exclusive and limits the range of choices, however, the term VoP is inclusive and is a more general term meaning the coding of voice into binary and transmitting the voice over a broader range of options—including Voice over ATM, Voice over Frame Relay, and Voice over DSL. By considering a wider range of VoP options, not just VoIP, an organization can often maximize use of its current networks, get around regulatory issues in certain countries, and extend the life of existing assets.

21

Chapter 2

Session Initiation Protocol (SIP) Early versions of VoIP used proprietary “homegrown” protocols to convert voice waves into bits and package those bits into discrete packets for transmission to a distant PC where they were depacketized and replayed over the speakers, and eventually the headsets of the distant computer. Early efforts to standardize the process led to the adoption of the H.323 protocol, really a family of multi-media call setup and media negotiation protocols, from the International Telecommunications Union (ITU), a global standards body under the United Nations. Eventually, a much more powerful multi-media control protocol, the Session Initiation Protocol (SIP), which can define not only voice but other types of multi-media sessions such as Instant Messaging and mobility, emerged from the Internet Engineering Task Force (IETF), which is the de facto standards body for the Internet. SIP is generally accepted as the primary standard protocol for VoP communications today and for the foreseeable future.

Internet Multi-Media Subsystem The Internet Multi-Media Subsystem (IMS) is an initiative of the next-generation wireless network architects intended to create a single operational structure that incorporates wireless and wired systems designed for multi-media communications. Although there are other, competing, approaches, IMS is a prime example of a vision for a single architecture that incorporates static, nomadic, and mobile multimedia systems.

Figure 2.1 is very similar to the diagram used to explain the benefits of taking a VoP approach. Analog telephony is the base of the family tree. The importance of starting with analog established that analog still exists in the network, and always will, because the endpoints that are communicating—humans—communicate by modulating analog waves and, at least for the foreseeable future, will continue to. The discussion of analog communication planted many seeds that would bear fruit later. For the traditional “Bellheads,” the members of the team who run the old phone network, the discussion afforded a nice walk down memory lane, jogging memories and knowledge all but forgotten, or at least not appreciated. For the “Netheads,” the discussion was the first of many reminders that there was much to learn about the world of voice if it were to be successfully grafted onto their IP network.

22

Chapter 2

Figure 2.1: The voice technology family tree.

Key points about analog telephony are that it works by translating continuous analog acoustic waves into continuous analog electrical waves and sending the electrical waves over a distance. Transmission distance is limited because the electrical signal becomes “tired” (the correct term is attenuated) over a distance and must be amplified to travel farther. The amplification process is not ideal because it amplifies the desirable signal (speech) as well as undesirable background noise. After a number of amplifications, the desirable signal is all but indistinguishable. Analog switching is also slow and problematic and analog systems have capacity limitations. Even so, analog is still used in today’s networks, even if just from a desktop or residential telephone to a Private Branch eXchange (PBX) or telephone company switch. One additional benefit of a review of traditional telephony was to provide a basis for understanding voice quality. Regardless of the many theories, metrics, and measurements, the true measure of voice quality has to do with how closely the analog wave arriving at the destination, after its trip through the phone network, resembles the wave that was sent. This basic fact is true regardless of whether the transmission system is analog or digital circuit or a digital packet system. In fact, an early objective called “toll quality” was based on the quality of the signal at the destination as judged by a panel of human listeners on a 5-point scale called the Mean Opinion Score (MOS) that has been imitated but never duplicated. After a thorough discussion of the analog era and the knowledge from that era that could be applied to the current project, we moved to the discussion of the digital circuit era. The comprehensive understanding of the digital circuit era that comes from operating a global digital voice network was fresh in the minds of the Bellheads because this is the network that they were operating every day and was the network from which they were attempting to migrate. The Netheads, however, got another reminder of how much they did not know about voice communications and how much there was to know. 23

Chapter 2 In this phase, we discussed the translation of analog waves, at or near the edge of the network, to digital bits using Pulse Code Modulation (PCM) or Adaptive Differential Pulse Code Modulation (ADPCM). We discussed the fact that, unlike voice coding techniques developed for VoIP/VoP systems that are optimized for human speech, PCM and ADPCM are analog-to-digital schemes that work for all sounds. In addition, although the VoIP/VoP systems would give a wide range of clever, bandwidth efficient choices, we can still use PCM and ADPCM in the VoIP/VoP architectures to maintain a higher voice quality. We also reviewed the global telephony network that existed within the organization and found that Private Branch eXchanges (PBXs), literally scaled-down telephone switches designed as the cornerstone of organizational voice networks, connected 64Kbps voice circuits to larger switches, which connected to high-capacity fiberbased connections formatted according to the rules of a standard called Synchronous Optical NETworks (SONET) in the US and Canada and a very similar system call Synchronous Digital Hierarchy (SDH) elsewhere. We also discussed the fact that regardless of the capacity of the SONET or SDH optical systems, they were nothing more than bigger and bigger bundles of the 64Kbps voice connections, at least in the older format in which they existed with this network. In the current network, voice was completely separate from data, video, and telemetry, and the multi-media bits never, ever, commingled within the system. Among other considerations was the fact that critical signaling information needed to establish and take-down the voice calls traveled in packets in a parallel network completely separate from the voice connections. On the plus side, the parallel signaling networks, governed by rules titled System Signaling 7 (SS7) in the US and Canada and Common Channel Signaling 7 (CCS7) elsewhere, were designed to be secure and virtually indestructible—without them, calls didn’t get connected and phone company revenue did not flow. On the minus side, were the costs of maintaining two complete networks, a circuit network for voice and a packet network for the signaling, just to set up voice connections. As we moved from our historical inventory to get a glimpse at the future, we had some area of common ground after which the IP folks began to feel more comfortable. The common ground was in the area of Voice over Fast Packet and its various branches. Unbeknownst to many, the global telephony standards organizations had begun work on a packet-based infrastructure for voice as early as the late 1970s with standards such as Broadband-ISDN beginning to appear in workable form in the early 1980s. The right side of the voice technology family tree (see Figure 2.1) is, in fact, representative of the work of the telephony traditionalists and includes a range of technologies from Voice over ATM to Voice over DSL, including Voice over Frame Relay and Circuit Emulation Services (see Figure 2.2). The left side of the family tree shows initiatives, as early as 1972, to put voice into discrete packets and send them over a shared network with data. These initiatives begin with little-known Voice over X.25 and VoIP over X.25 and include the much more recent, and more memorable, VoIP over the Internet projects in 1994 that were the true ancestors of VoIP as we know it today.

24

Chapter 2 Until 1995, however, progress in packet-based voice was interesting but not of commercial interest. It was, in fact, very much in the hobbyist realm, as we might think of ham radio, for instance, until the invention of the packet-circuit gateway in 1995 made it possible to call from a PC to a traditional telephone. It was not long after that calls from traditional telephones to PCs were demonstrated and then, the service that proved most disruptive to the established traditional telephony companies, calling from a traditional telephone to another traditional telephone completely transparently over an IP-based bypass network. This is where the fun began and this is, precisely, the objective of the VoP migration project: to gain the benefits of a combined voice, data, and video network and bypass traditional telephone carriers for all internal and some external calls while maintaining connectivity to the traditional telephone network to be able to reach persons, be they on stationary traditional telephones or the growing population of mobile phone users, as and when needed. In addition to the raw technical arguments for or against a specific packet voice option, experience and analysis showed two more important dimensions: legal/ regulatory and operational. From a legal/regulatory standpoint, it is interesting that packet voice is not legal in many of the jurisdictions in which this global organization operates and, even though there has been a broad relaxation of anti-VoIP statutes, there are still nations in the world where VoIP, or at least non-circuit voice communications, breaks certain statutes, violates specific tariffs or contracts, or is out-right illegal. In these cases, the business and technical arguments have no weight against the legal ones and legal and regulatory compliance is still a check-off for any change to the telephony status quo. As far as the operational dimension of VoP systems, the technology choices were weighed against voice quality, delay, and delay variation issues, price, multi-plexing efficiency and system complexity and some interesting relationships emerged (see Figure 2.2). Overall, voice quality was deemed to be highest in the systems in which all aspects of voice transmission could be most strictly controlled—that is, in circuit systems. The less control and the greater the influence of other traffic, the lower the quality. Considering delay and delay variation, the further you go away from circuits, the more delay and delay variation. Considering these items only, it would appear that circuits are ideal and packet-based systems inadequate—that is, until price and multi-plexing efficiency are factored in. VoIP systems that use bandwidth-saving silence suppression and overhead compression schemes to provide the best multi-plexing, or sharing, with data and video have the lowest overall costs though they are somewhat more complex to operate than circuit-based systems.

25

Chapter 2

Figure 2.2: Analyzing operational dimensions of VoP systems.

Multi-Protocol Label Switching (MPLS) Multi-Protocol Label Switching (MPLS) is an approach to networking characterized by this author as “the best of both worlds.” MPLS is a way of delivering Quality of Service (QoS) in Virtual Private Networks (VPNs), the secure package in which many organizations are purchasing network services, by allowing a dynamic choice to be made whether to route or switch information traversing a network. Routing, the traditional Internet way of sending information across a network, has the benefit of not having an initial delay to establish a path from end-to-end across the network, but each packet must be analyzed and forwarded individually. Switching, in contrast, the traditional way that telephone networks work, does have an initial delay but whisks each subsequent bit quickly to its destination. The real answer to which approach is best cannot be answered until we know what information is being sent and, more specifically, if it is a long-lived or short-lived flow. For a short-lived flow, something like a small Web page, for instance, routing is best because each packet is forwarded on its own: there is no need to go to all the trouble and delay of setting up a switched path for such a small amount of information. A long-lived flow, however, such as a longer VoIP conversation or a video over IP program, will benefit greatly from having a pre-established path and not having to go through the requisite packet analysis and forwarding of each packet. MPLS is “the best of both worlds” because MPLS is capable of routing all traffic until it has determined whether a flow is long-lived or short-lived and setting up a switched path for those flows that turn out to be long-lived. Early implementations of MPLS would always route over the available IP routing mechanism and switch over Frame Relay, but newer variants are capable of switching over everything from ATM to Wave Division Multi-Plexing to SONET/SDH to dedicated fibers, including Frame Relay and the IP RSVP bandwidth reservation protocol.

In the case of the global energy giant, the impact of this analysis was felt more on the prioritization of systems to migrate and a heightened comfort level with leaving some non-IP systems in place for a longer period of time during migration, but had no appreciable impact on the ultimate objective, which was still to eventually have an all-IP multi-media architecture. This analysis did, however, support their decision to migrate much of their global IP traffic, both voice and non-voice, over to a managed MPLS-based VPN to get the combined benefits of IPbased applications, QoS, and security offered by such VPNs.

26

Chapter 2 In other instances, however, a broader look at VoP caused several large organizations to rethink their VoIP objectives. In many cases, including a North America-wide system for a major manufacturer of industrial tools, the organization realized, prior to embarking on a traditional VoIP approach, or early enough in the cycle to retreat and regroup, that they did not really need the power of multi-media or even magic of SIP: what they truly desired was less-expensive voice communications at a level of quality comparable to, or just less good than, their traditional dial network. These organizations often implemented Voice over Frame Relay or, less frequently, VoDSL or Voice over ATM. Impact of VoP Having considered VoP almost in isolation, the next step was to consider the impact of VoP on the existing network. The existing network was a predominantly data network with a rapidly growing set of video services. The data, however, includes the traditional menu of services— browsing, email, file transfer—and the movement of gargantuan volumes of near-real-time telemetry data that most network operators do not have to deal with. The initial “VoIP is just another IP application” mantra was quickly thrown out in favor of the “VoP is a unique application with its own special requirements” mantra. One reason for this change of mantra is that early, especially residential and consumer flavors of VoIP, used very low-bandwidth compression algorithms which, while bandwidth efficient, provided a lower quality, or at least a different quality of voice. Many businesses have opted to continue to use the traditional Pulse Code Modulation (PCM) voice coding scheme even after their migration to packet voice. Keeping PCM end-to-end, regardless of whether the bits are traveling in circuits or packets, assures better compatibility between different systems and reduces the number of translations between different coding schemes, which also improves voice quality. What this means in operation is that instead of 6 to 8Kbps per voice connection, each connection consumes 80 to 110Kbps of bandwidth, though the actual utilization may be considerably less if silence detection systems are used to not send packets containing silence. Another impact consideration is that voice systems can provide less delay, less delay variation, and less susceptibility to packet loss by putting fewer voice samples in each packet. Although this approach does have its benefits, as stated earlier, it also has the drawback of taxing routers and transmission systems by requiring a lot more work for a lot fewer bits being transmitted— and all this is going to have to be considered when assessing the true impact of adding telephony to the network.

27

Chapter 2

VoIP and IPT Return on Effort The financial measurement of Return on Investment (RoI) is a calculation of the financial benefit achieved for a specific investment of capital. An additional metric that takes into account intangible—and, yes, admittedly, soft/strategic benefits—is Return on Effort. RoE had no value to the global energy company, but many organizations find a consideration of RoE a good way to take into account aspects of a move to a VoP system that cannot be captured any other way. For instance, in one of the case studies reviewed in Chapter 1, Heritage Bank, the financial RoI calculation was sufficiently terrible as to scream to anyone who would listen “YOU SHOULD NOT HAVE DONE THIS!” but the RoI calculation neglected the fact that the system being replaced by the new VoIP system was no longer available from the manufacturer, spare parts were no longer available, and that the bank had the choice of a forklift upgrade or ceasing to add staff, branches, and new clients. For the bank, as for many companies adopting VoIP, it is more expensive than the current system and an upgrade does not make financial sense, but there is no other option. RoE is also a way of identifying other intangibles, such as the school district that was able to provide in-room phones for teachers where none had previously existed or the pharmaceutical lab that was able to display employee and contractor photos on phones for enhanced security.

Enterprise Telephony Migration At this point, it has been determined that the organization will migrate to a new packet-based telephony system. What lies ahead is the inevitable budget process, the migration itself, and, once across the bridge, the IPT life cycle of planning and assessment, pre-deployment testing, ongoing operations, and optimization. The Budget Process For the global energy company, as for many organizations giddy with excitement at the prospect of finally being able to retire traditional voice assets and headcount and realize the benefits of “free voice,” the excitement disappears quickly as the budget process begins. The first realization was that the skills of the traditional telephony people are still in demand and, in fact, are becoming more unique and valuable than their IP counterparts due to the natural attrition of the traditional skill sets. In the case of the global energy company, and many other companies, the following type of plan was anticipated—as soon as the first packet carrying voice flowed from one IP phone to another, the un-needed old telephony regime could be shown out of the building and their cost saved. What actually happened was that the best of the traditional voice and IP team were kept on staff and the less useful, less-skilled individuals from both groups were terminated, though nowhere near 50 percent of the staff was cut. For this situation, it turned out that it took about 70 percent of combined prior staff to run the new, consolidated network, though it was felt that cross-training over time to create a combined data/voice/video multi-media support tech could provide greater than 30 percent savings on staffing costs.

28

Chapter 2 The second realization was that packet voice requires special hardware. The new hardware would take the form of either new, specialized phone devices—at roughly the cost of earlier digital and PBX phones—or gateway systems. The gateway systems would be needed to adapt the traditional telephone sets to the new IP connection. In most cases, some of each new system would be required, at least through some migratory period. This is an additional, and often unexpected, financial burden on the organization, but a surprise that can be avoided with good planning. Another related surprise that can be avoided is the fact that new phone sets must be provided with AC power or powered via Power over Ethernet (PoE), both of which require additional planning and costs. In some cases, organizations spend as much, or more, than they spend on the new voice over packet system to provide power to the phone systems because, unlike their analog predecessors, the new systems rely on “wall power.”

The third realization is that, like their predecessors, IP-based multi-media systems are complex systems that require management, measurement, troubleshooting, and problem resolution—in effect, life cycle management at each step through their operational cycle. In fact, the management required by traditional telephony systems is often hidden “behind the scenes” because dial tone has often been purchased as a service rather than a bunch of piece parts to be assembled by the end user organization, such is the shift in thinking from telephony as a service to telephony as a managed application. But do not be fooled, there are important management functions that must be completed routinely at each step of the life cycle as will be described in more detail later. Reuse of Existing Assets In the ideal situation, existing assets can be reused. This is true of human assets that can be retrained or cross-trained in the needed disciplines. Hardware assets can often be reused as well, with the addition of a translation system or gateway. One benefit of gateways is that they can provide access to voice connections on existing networks as well as special features provided via the Advanced Intelligent Networks (AINs) that traditional carriers have built over the years. The physical positioning of the gateway also impacts the cost and ability to reuse existing assets as well as the ease of support of the new system. Consider Figure 2.3. It might be possible, for instance, to assess the needs of the user community and divide users into three groups: those users who need new IP capabilities now, those who will need IP capabilities at some time in the future, and those who simply need traditional dial tone.

29

Chapter 2

Figure 2.3: Determining which assets can be reused.

The first group would be provided with new IP phones and directly connected to the IP network. Because this group is a subset of the entire user population, however, the cost of the new phones is a fraction of what it would be to provide new phones for everyone, and the resource impact on the IP network is also a fraction of what it would be if all users were migrated. Some scheduling algorithm could be applied to the second group to determine at what point they would be migrated to the new IP system, and the third group could simply be provided with “personal gateways,” often called Analog Terminal Adapters, which would allow their traditional telephones to be connected to the IP network. They would still use bandwidth on the IP network, but the hardware cost and complexity of putting this set of users on the network would be substantially lower than moving all users to new IP phones. At this point, the overriding consideration might well become the calculation of the cost of maintaining the old traditional network just to support those users who had not yet needed to migrate, and it may be advisable to complete the migration sooner, rather than later, and retire the old network. However, if a larger gateway was positioned on the network side of the PBX or premises switch it is possible to reuse all the premises phone systems and wiring and attach all existing telephone users to the IP network via the gateway. This may be an interim step until a local shared IP softswitch is implemented or until a remote IP-attached softswitch is implemented. In any case, it is an approach that can be used to extend the life span of the premises voice solution. New Assets Needed In any case, new assets will be needed in order to extend the life of the old system and to bring new features and capabilities to the users. New assets will include, at minimum, hard or soft phones, gateways, softswitches, MPLS-capable routers and MPLS-aware end systems, Virtual Private Networks (VPNs) built with those tools, and additional bandwidth. New assets will invariably include new management, monitoring, and troubleshooting tools as well as tools to ascertain compliance of services with Service Level Agreements (SLAs).

30

Chapter 2 Assessing Urgency and Prioritizing Migration Except in the smallest, single office situations, it is not advisable to flash cut all telephony users to a new system. For this reason, it is necessary to assess the urgency and prioritize which areas of the organization will be migrated in which order. As difficult as it might seem, political pressure is the last thing that should be considered when determining the order of priority for migration. In fact, the most visible or boisterous departments should be migrated last to assure a minimum negative impact on the overall migration. Migration order should be based on more reasonable considerations: •

Which office or region has outgrown, or is closest to outgrowing, their existing system?

•

Which region has bandwidth or IP network infrastructure good enough to carry voice, and other resources to accommodate the new telephony upgrades?

•

Which region has the most technically savvy support staff?

•

Which region was involved with early testing and/or staging and is, therefore, most familiar with the new system?

•

Which office or region is closest, because the earliest migrations are always learning experiences and should be expected to go slower and be more problematic?

Migration Migration can be divided into three categories: avoiding migration, partial migration, and full migration. Within the partial migration and full migration categories, the roll-out may be phased based upon a variety of factors, as will be discussed later. Avoiding Migration Avoiding migration is usually “avoiding migration for now,” which means that the office, division, or region under consideration either has a recently acquired system that has not yet been depreciated or that the group will be sold or otherwise liquidated before it is required to join the new VoP network. One strategy adopted by the global energy company that not only eased migration but also stretched the use of their assets, thus reducing their overall budget, was to establish regional refurbishing and staging centers for equipment that had been taken out of service as a part of the upgrade. As an example, let’s say that an analog key system in Tokyo no longer had expansion capability because the upgrade components were no longer manufactured and that this is the only reason why Tokyo was being upgraded to a new VoP system. Sydney had exactly the same issue and the same key system. In this case, Tokyo might be upgraded to the new system, thereby freeing components that could be used to upgrade Sydney and avoid migration to the new VoP system, at least for now, or vice versa. If there were no immediate need for the components removed from Tokyo, they could be refurbished and placed into an upgrade inventory for the time when they would be needed to upgrade a system and avoid migration.

31

Chapter 2

Partial Migration Partial migration is accomplished as previously discussed by using strategically placed gateways to allow existing assets to be kept in place, thereby extending their useful life, especially where only a limited subset of the new VoP capabilities were needed by the office or region being partially migrated. Full Migration Full migration means that all non-VoP components are removed and replaced with their VoP counterparts. Delaying VoIP/IPT It might seem a bit late in the process to ask this question, but can a move to VoIP/IPT be delayed? One reason why it is not too late in the process is that up to this point all the planning exercises are what military planners call “tabletop exercises”—no changes should have yet been made, no systems implemented. Everything up to this point should be on paper, but not yet realized in the real world. Thus, what are the pros and cons of delaying a move to VoIP and IPT? If there are no competitive or operational pressures to make the move, the organization should consider delaying the move. Each day that goes by, prices for components and services get lower, standards solidify further, and the various aspects of implementation and operations are better understood. For each day that passes, traditional telephony components are further depreciated, and for each day that passes, there is one more day without the potentially unnecessary disruption to operations that virtually any change, especially the change to a system as fundamental as your phone system, can bring. It is only prudent to stop at various times in the planning process to revalidate your objectives and to consider any change in business climate that would mandate either slowing down or speeding up the migration process. Systems Integrator vs. Do It Yourself Once the decision has been made to move to VoP and it has been determined the extent to which it will be rolled out in the organization, it is necessary to confront the issue of phasing and project management. Some organizations choose to migrate entire departments or offices to VoP, while other organizations will phase-in VoP by job function—for example, phasing in support staff first to allow them to gain experience with the technology. Once the phasing has been determined, a project management schedule can be developed. It is at this juncture that many organizations realize that they do not have the resources to do the task. Smart organizations realize that if they are resource-constrained before a VoP rollout project, they must line up additional resources before they can begin their VoIP rollout. Some organizations attempt to be clever and hire and train college interns or other “temporary” workers to assist during the rollout. Although the concept is right—workers will be in high demand during the rollout but will not be necessary after migration—the approach is wrong. A key consideration in this scenario is that knowledge gained during the rollout should be captured in some way because it is valuable to the support process. For this reason and others, many organizations consider engaging a systems integrator to assist in the rollout.

32

Chapter 2 When an organization weighs the systems integrator vs. DIY approach, the organization should remember that the VoP migration is not simply a matter of plugging VoIP phones into existing Ethernet ports and making phone calls. Instead, it is a highly complex, and highly visible, project within the organization. The systems integrator should be evaluated based upon demonstrable success, site visits to the systems integrator, and interviews with project managers and other key persons. In addition, the systems integrator should be involved as early as possible in the planning phase. It is also important to consider the systems integrator vs. DIY debate as early in the migration planning as possible so that the systems integrator may be considered in the budgeting process. Although it is not possible to predict prices without more detail about an individual project, it is not uncommon for an excellent systems integrator, who can assure project success, to cost between twenty and thirty percent of the overall price of a project. Managed Service vs. Do It Yourself For small and midsized organizations, voice has largely been a “managed service” that has been provided by a telephone company with the occasional small to medium enterprise venturing into the “do-it-yourself” voice business. Large to very large organizations have been quite the opposite and many have, since the late 1970s, provided their own dial tone via PBXs or technical arrangements with similar names. As we move into the multi-media era, it seems as though many organizations are viewing VoP as a new application for their IP intranets and Internet networks and, therefore, increasingly as a do-it-yourself project. But, this does not need to be the case, and, in fact, the migration to a packet-based voice system may very well be the opportunity to rethink the way voice is delivered within the organization and between the organization and the outside world. If an organization does decide to choose a managed service approach, they can ignore many of the technical details of the inner workings of the system and instead “take the temperature” from the outside and ask the doctor how the patient is doing. This can be accomplished using service provider-offered tools that provide a snapshot of the network or even embedded measurement systems in the customer’s own equipment that might give a more accurate, or at least more timely, report on the health of the overall system and its individual connections. In any case, these tools provide nothing more than a second opinion that might—or might not—correlate with the health ratings offered up by the service provider. If an organization does decide to take the do-it-yourself approach, the implementation of VoP services will require the use of measurement and monitoring tools that reveal far more “personal” results—results that not only indicate how the system is performing but also what the organization needs to do to repair any problems or, hopefully, the level of pride they may take in their own good results. In either case, the measurement and analysis tools are useless without baseline measurements and a set of rules that typically take the form of a document called the SLA—and even though the SLA for years has been thought of as a way of assuring compliance of the service provider, and it still is, the definition of service provider probably needs some updating. In the managed service scenario, the service provider is still who you think it is—“them”. But, in the do-ityourself scenario, the service provider has become “us,” because you are now an internal service provider and the internal departments, divisions, regions, and employees are the consumers of this service—they are the customers. 33

Chapter 2 A further consideration is the interplay of multiple SLAs from multiple parties. Take, for example, a scenario in which a managed service provider is providing VoIP service on a MPLS VPN provided by a separate provider. In this case, the two SLAs are inextricably interwoven. Can you penalize the VoIP service provider for not connecting calls on an underlying IP network that cannot connect across the MPLS network, or if the MPLS network does not meet specific QoS metrics? And what if there are multiple MPLS networks provided by multiple providers? This is an issue often seen with larger, multinational organizations. These items must be considered to achieve a reliable and fully functional VoIP implementation. It is also possible that these items have already been considered, and resolved, in some other networking context, so it is worth researching before diving in. Although the network is evolving into a complex creature delivering real-time, near real-time, and non real-time voice, data, and video, the SLA must still be written in terms of metrics that are meaningful to the specific services being delivered by the network to the user/consumer. Let’s take a closer look at this critical management tool—the SLA. SLAs One of the key items that will require revisiting, and refreshing, is the SLA. The SLA is a wonderful tool the full potential of which has never been realized. In its most basic form, the SLA is a contractual vehicle which, at the very least, documents the minimum level of service that a customer is willing to accept before they are due some penalty or refund on their service, but it can, and should, be so much more. It can, and should, also provide a description of the actions taken by a carrier or service provider should the user not keep up their part of the arrangements, such as using a service class for which they have not contracted. One possible model that might emerge is that transport becomes a commodity over which differentiated services flow. In this model, the prior performance of various transports, representing some combination of different Class of Service (CoS) offerings from one or more transport providers, is known and the closest CoS to that needed by a given service at a moment in time can be selected, possibly on a “pay as you go” basis. An alternative to that would be that bids would be requested, in real time, from available transport services to provide a certain CoS at a moment in time. How would this work in practice? Simply recall the example in Chapter 1 of the call from a Milan coffee shop. Penalties Most SLAs are lopsided, especially when it comes to penalties. In most cases, the SLA is written in such a manner that any shortcomings on the part of the service will result in penalties from the service provider. However, the method of proving that penalties are owed is notoriously difficult and only the most exacting and picky end user organizations routinely see any penalties. Both of these problems must be addressed.

34

Chapter 2 SLAs should impose a penalty on the service provider (be they external providers or an in-house group) if they do not perform; in addition, SLAs should impose penalties on the user organization if they don’t perform, such as if they continually ask for bandwidth beyond the contracted rate or impose a special availability requirement for a certain location. Finally, SLA compliance must be automatic. At the present time, too few carriers and service providers have embraced the SLA as a strategic customer relationship management tool and require the customer to jump through hoops to get the penalties they are due. SLAs and the compliance information supporting them should be available to the customer when and as needed. We will see a lot more of this in the multi-media future, and sophisticated monitoring and management tools will be required to keep customers apprised as to the QoS delivered by a network versus what was promised and contracted. In fact, the educational process that goes along with this effort should be initiated as early in the process as possible because the process of educating the user on SLA compliance is also the process of educating the user on how to select multi-media services without relying strictly on price. True Impact of SLA on Business and Operations A fully integrated and automated SLA-based network monitoring and management system can provide both long-term and “canary in the mine” types of visibility to overall and service specific operations. In an environment of enlightened collaboration between user and service provider, the SLA is a key tool to successful operation and a path to continuous improvement. SLA metrics can, and should, be modified over time to reflect actual network objectives and realities.

The IPT Life Cycle The four phases of the IPT life cycle are planning and assessment, pre-deployment testing and implementation, ongoing operations, and optimization. These four phases are discussed in this section.

Figure 2.4: The IPT life cycle.

35

Chapter 2 Planning and Assessment The objectives of planning and assessment are to develop a replacement system that initially delivers replacement functionality and eventually adds important strategic functions. The migration process is characterized by a new, fresh commitment to traditional telephony characteristics such as reliability, security and fraud control, and customized services deployment as well as making desirable new aspects, such as a multi-media focus, available organization-wide to all users who need them. All this must be taken into account in the planning and assessment phase. It can also not be stressed enough that management expectations must be set so that a sufficient amount of time and resources can be invested in the planning and assessment phase to make the phases that follow it successful. VoIP is not just another application that can be launched by plugging in a few IP phones. A proper planning and assessment phase will take all of the needed factors into account. To be successful in your next-generation network venture, expectations should be set as follows—only in so doing will the results you achieve have any hope of being judged a success when compared with the expectations you set: •

Users—Different is not bad. The new system will initially provide all the important functions of our current system and then, after the baseline is established, and the migration is complete, will provide important new capabilities. The system will sound different but this is not necessarily a bad thing. Our system voice quality will be ____________(not as good as, nearly the same as, better than—fill in the blank based on your objectives) and the new system will be more flexible and give you more freedom while allowing you to stay connected.

•

Management—The cost of our new network, when taken as a whole, may be the same or higher than what we are paying now, but we must move forward to assure the continuity of our voice communications and eventually acquire advanced capabilities of strategic benefit, though our unique combination of geographic coverage and scale may allow us to realize overall raw cost savings. There will be four specific phases, each with its own specific objectives. Investing more time, effort, and attention in earlier phases will lower the overall costs and increase the value to our organization. The phases are planning and assessment, pre-deployment testing and implementation, ongoing operations, and optimization. Contrary to popular belief, VoIP is not “free voice” and VoIP is not “just another application.”

Organizations adopting multi-media will not, in most cases, actually ‘move’ at all. In most cases, they will already be providing or using traditional and/or VoIP-based systems and will simply transition to multi-media as non-disruptively as possible. The first step is to ascertain the current services and features that are being utilized to assure their accurate reproduction in the new system. The next step is to move as quickly as possible through the transition phase, though many organizations would be well advised at this point to skip the VoIP step and move directly to the ultimate converged IP-centric multi-media IPT phase. This step avoids the risk of being in the transition period and reduces negative user impact. A critical aspect of planning and assessment is ensuring that the new system performs at minimum the key functions of the system being replaced and that transitioning to the new system does not present any unexpected and unpleasant surprises. One way to do this is to do a functions audit on the old system in the planning phase and even prior to acquisition of a new system as a precursor to defining mandatory functionality during procurement. 36

Chapter 2 The obvious approach would be to ask the systems administrator or manager of the old system which functions must be supported, but the result of this request usually produces an operations manual or sales brochure listing all possible functions. It is not desirable, for numerous reasons, to attempt to replicate all functions of the old system. A reasonable middle ground is to print colored cards with all the key functions from the operations manual and a blank for “write your function here” and distribute the cards to the users. When a telephony user performs a specific function, they write their phone extension number or other identification and check off the function performed. This would allow an audit of used phone functions to be performed. An alternative to this would be if your PBX, phone switch, or phone service had a reporting function that identified, at minimum, which features were programmed and, at best, how often the function was used and by which users. After the audit, it might not be possible to guarantee that the new VoP system will perform all the functions of the traditional system, but it will be possible, in advance of migration, to identify potential problem areas and workarounds. Pre-Deployment Testing and Implementation The objectives of pre-deployment testing and implementation are to assure the outcomes from the planning and assessment phase and to determine the impact of adding the new applications to the existing network—and then to rollout the new system into the organization, assuring that the systems are operating in compliance with the SLAs. The use of sophisticated, specialized modeling and planning tools in the first two phases will allow you to assure the maximum benefits in the shortest amount of time with the least possible disruption to your operations. Ongoing Operations and Optimization Onging operations and optimization are two phases with opposite objectives. It is the objective of ongoing operations to assure consistent functionality across the network, regardless of where an individual is in the organization geographically. Ongoing operations is constantly measuring the network’s performance against established benchmarks to assure unwavering compliance and absolute consistency of network services with SLAs. Optimization, in contrast, has as its primary objective improving operational aspects of the network services and adjusting the operational baselines to reflect new needs and requirements. Examples might be to improve voice quality and update SLA levels, which are operational improvements, or to decommission PBXs or IP gateways, which will provide a cost benefit. Although ongoing operations is about consistency, the optimization phase is about a regular, constant, ongoing program of positive change in network operations. Optimization takes the measurement and SLA compliance statistics gathered during ongoing operations and seeks ways to improve those statistics with specific target outcomes. Desirable objectives will include financial or operational goals. The process will involve conceptualizing improvements and validating assumptions using modeling or simulation tools and then testing them in a laboratory or isolated network segment before implementing the changes in the actual network. After network performance has been assessed and any needed fine tuning done to the changes, the network operational benchmarks are adjusted and new optimization goals are established.

37

Chapter 2

Managing the IPT Life Cycle Now that you have a better understanding of what the IPT life cycle encompasses, you must now develop a better sense of how to manage the life cycle. This guide will provide much more insight into the management process in later chapters but here will take a look at the tools that must be in your toolkit in order to meet the challenge. Enterprise-Accessible Tools If you have opted for the “do-it-yourself” approach, you will have access to sophisticated tools embedded in the routers, switches, gateways, end systems, and other components of your networks. Even if you haven’t, there are still a rich set of tools that can be used to provide “the big picture” as well as pinpoint areas that require a more granular focus. MSP-Provided Customer Network Management Managed service providers (MSPs) allow end-user organizations to view thin slices of their network: the slices that are important to the transmission of the customer’s information, masking information about other customers and providing a true VPN view. This capability is called Customer Network Management (CNM)—allowing the customer to see their part of the shared virtual network—and has been around in some shape for well over two decades. Early CNM systems were periodic summary snapshots showing not-so-very-near-real-time (NSVNRT) statistics that bore little resemblance to the real world or the SLAs, while more current systems, especially those based on MPLS VPNs, can provide darn-near-real-time (DNRT) views down to a connection level and do, in many cases, allow customers to adjust certain characteristics, such as CoS or available bandwidth, in real time. Although not ideal, CNM is a valuable tool for enduser organizations that must rely on third parties to provide and manage their service. Manufacturer-Provided Systems Manufactures provide the most accurate, most granular, and seemingly most desirable systems embedded within their hardware. The tight coupling of hardware and operating software provides the finest possibly granularity of analysis but, in fact, tends to be limiting and confounding unless the entire network, from the physical layer transmission systems to the application servers themselves, are provided by a single manufacturer and, of course, that is rarely the case. The proprietary, manufacturer provided, tools do have their place, when optimizing a network, for instance, or at higher levels of escalation for system-specific trouble shooting, but lack the broader, more standardized and less platform-specific capabilities of systems provided by third parties. It is also true that third-party systems lack any vendor bias and can be counted on to be more impartial when a system is comprised of components from multiple manufacturers or as a good “second opinion” in those rare cases that a system is constructed completely from the components of a single manufacturer.

38

Chapter 2

Third-Party Tools Third-party tools for modeling, monitoring, measuring, and managing are invaluable in that they provide broad, cross-platform visibility into aspects of network operation that far exceed what humans alone can do without those tools. The consistent and proper application of these tools at various stages in the life cycle of the multi-media network spells the difference between success and failure. The selection of these tools is the subject of the third-party tools selection report card described in the next chapter and their application to the life cycle of the network are described in further detail in subsequent chapters. Importance of Modeling, Monitoring, Measuring, and Managing As important as the right tools to dimension and manage emerging multi-media networks are, they seem to go against conventional wisdom about how a multi-media IP network is implemented and managed. Time after time, the biggest and most visible example of IP networks today, the Internet, has been described as “plumbing,” or a common conduit for any type of information, and the term plug-and-play has been used over and over to the point that it has become ingrained in the subconscious, and almost into the genes, of ‘Net users and managers who must make budgetary decisions. “Free Voice” is another obstacle that must be overcome in order to be granted the funds needed to operate a network that is, consistent with another metaphor, “the nervous system of an organization.” What management must understand, and subsequently follow with appropriate funding, is that the multi-media IP network, be it Internet, intranet, or extranet, requires special tools to build and maintain its health. Experience has shown that an initial cost of 5 to 7 percent for procurement and training, with an ongoing 10 to 12 percent of operating budget dedicated to staffing, ongoing training, support, and network optimization will pay substantial dividends in network reliability and cost reductions. Proper funding of personnel and systems, will ensure that organizations reap all the rewards of their move to a new voice platform. Put in its proper perspective, the shift from traditional telephony to its packet-based replacement is like the differences between a 1967 Chevy and a 2007 Lexus. Both have four tires and a steering wheel and are designed to get you from one place to another. But, what is “under the hood” varies dramatically. Your ’67 Chevy was so simple that Floyd down at the service station could pretty much provide any service you needed. In fact, the term “keep it humming,” relative to automobiles, came from the fact that much of the diagnostic process involved listening to how the car sounds. The VoP system is much more like your 2007 Lexus. It requires specialized service at a Lexus dealer or service point that is capable of utilizing the diagnostic interface, and Floyd at the service station wouldn’t have a hope of understanding the sophisticated, computerized guts of this vehicle. If you have an accident, a signal is sent to a customer care center that calls you and will dispatch 9-1-1, an ambulance, and service vehicle within seconds, even if you are incapacitated. And what about locking your keys in the car? Well, the car can be unlocked from the satellite, as well. As much as the Lexus may resemble the Chevy in form and function, it is a fundamentally a different creature with different capabilities, different management requirements, and different maintenance needs. So, too, is the case with VoP and traditional telephony.

39

Chapter 2

Summary This chapter has begun the practical job of looking at the key elements that further set the stage established in Chapter 1. It has also begun looking at the key phases of the life cycle of the packet-based telephony system and how to harness your knowledge of the natural rhythms of that system. The next chapter will continue to dive deeper into the life cycle. As promised, the tone and intent has shifted from one of establishing a baseline understanding of some broad, abstract concepts, to providing specific, actionable items that you can perform for your own situation. The next chapter will continue this approach.

40

Chapter 3

Chapter 3: Planning and Assessment At a professional car race, the cars are powerful, fast, and full of muscle, zipping around the track at amazing speeds; the drivers defy death and beat physics at every turn. What only the true racing fanatic realizes, though, is that there is more to this feat than the power of the car and skill of the driver. Considerations such as the preparation of the track, the track surface, the inflation of the tires, the bank of the turns, the surface temperature of the track, minor variations in the fuel mix, and other seemingly arcane considerations contribute in a very real way to the outcome of the race. This chapter talks as much about the track—the high-speed broadband surface over which telephony and other multimedia content races—as it does about the vehicles—traditional and multimedia phone calls. This chapter will also delve into some of the seemingly arcane aspects of both, making good on the promise in the first chapter to educate phone people about IP networks and IP people about telephony systems. The first two chapters introduced some of the principles driving the change to VoIP and IPT and noted that this guide will challenge the market “buzz” that has been generated by emerging and evolving packet telephony technologies. This chapter will continue exploring both the pros and cons of this technological shift, which should be more of an evolution than a revolution. Planning and assessment must begin with a careful identification of the business drivers for change, which will, in turn, provide objectives that will guide the project and allow the measurement of the outcome. This evaluation must include thoughtful analysis of the current methods of voice communications and both the voluntary and involuntary drivers. Voluntary drivers include the desire to reduce cost and increase productivity as well as advances in business applications that require the integration of new technologies. Involuntary drivers can be factors such as manufacturer discontinuation of existing systems and an enterprise’s lack of desire to continue investing in hardware that is rapidly becoming obsolete. Planning and assessment also includes setting proper expectations with both users and management. Expectation setting includes defining proper design and ongoing operational criteria, such as technical requirements and acceptable operational thresholds. Important metrics for all telephony applications are identified, whether for simple two-party telephone calls, voice and/or multimedia conferencing, voicemail and presence applications. Methods for translating those metrics into a measurable and actionable Service Level Agreement (SLA) for internal or external service providers are also provided—whether those providers are carriers and service providers who supply the track or service providers, systems integrators, and inside entities who provide the vehicles that run on the track. This chapter will also include a unique, comprehensive capabilities inventory that is designed to ensure that every capability used in the current/old system is at least considered for inclusion in the new system.

41

Chapter 3

Business Drivers for Change When companies explore Internet Protocol Telephony (IPT) solutions, the focus must be on key business areas because it is business applications and needs that will drive change, and the meeting of business needs that will define the project and make the project successful in a business context. As different as it may often seem an IPT project is no different than any other project that impacts the way in which every member of an organization does their day-to-day job. In fact, the IPT project touches the lives of every member of the organization and the way in which they communicate with each other and with those outside the organization. When evaluating the full range of business needs it is vital to the success of an IPT implementation that management and user requirements are assessed and proper expectations set. It is also important to fully understand all the business drivers behind the shift to IPT. If cost reduction is the primary driver, there will be different factors to consider than if the primary driver is deploying new, converged applications. Understanding the key drivers for the project helps maintain focus on those factors that ensure success. Communicating the objectives clearly with everyone involved, including suppliers and service providers, helps maintain a clear view of the expected results and allows all parties to weigh in on the viability of the objectives. When the migration from traditional telephony to VoIP and IPT began in 1995, businesses and industry pundits alike projected incredible cost savings. These savings were primarily seen as a natural outgrowth of consolidation. The consolidation to a single physical network and single cable plant also led companies to envision a consolidation of staff, reducing the support requirements for a single converged network. Much of the punditry surrounding convergence neglected the inherent complexity of telephony itself—for years the market had been lulled into a false sense of simplicity by the fact that anyone could lift the receiver on any phone, press the correct series of digits, and—voila—a person anywhere in the world would lift the receiver and answer! Over time actual experience with VoIP has made a mockery of most claims of deep savings vs. traditional telephony, unless, of course, users are willing to abandon most features of traditional telephony, which is hardly a fair comparison, and use the nearly free PC-based systems. Although the savings of VoIP circa 2007 are substantial compared with the cost of traditional telephony circa 1995, VoIP today often demands a premium as older, depreciated gear with a service life in the range of two decades is replaced with an unknown or 18-month to 5-year expected service life. And one must also consider that purveyors of traditional telephony have reduced the prices for use of their reliable, high-quality traditional systems. This leaves issues such as manufacturer discontinuation, including both the discontinuation of lines of traditional equipment as well as the discontinuation of the manufacturers themselves, and the desire for new, integrated, multimedia applications in the forefront of reasons to purchase the new, often more expensive, VoIP and IPT systems.

42

Chapter 3 As networks converge, new integrated services can provide unique, new business processes. In a fully integrated network, the tighter relationships between data systems, telephony services and video lead to new opportunities. Take customer service as an example. Customer data, collected over both telephone and computer systems, can enable new methods in providing customer service, possibly at a lower cost, but more importantly in a manner that directly increases customer satisfaction, and indirectly increases revenues, enhances an organization’s reputation, and raises the bar for customer service within the industry; thus, putting pressure on competitors to meet the new challenges or lose market share. Call center functionality, for instance, can easily be dispersed geographically, and not just through low-cost VoIP-driven off-shore call centers but also through distributed, call-center-free call takers working part time at home on a demand-driven basis at a lower total cost but a higher rate per hour. It’s the technology that makes it easy to integrate teleworkers at remote locations, at lower costs and providing higher levels of service while better meeting the personal needs of the workers. An integrated, Web-based customer service application can handle both telephone calls and provide Web services such as interactive online and TTY for the deaf. Those users who would prefer to communicate via their browsers can explore a business Web site and click a button for customer support that places a VoIP phone call to a company representative who is working at home or in a centralized call center. With IPT in a fully integrated environment, computer telephony integration (CTI) tools can provide the customer support representative with comprehensive information to assist the customer. This information might include not just customer contact and account information but also a screenshot Web page that the customer last visited and a complete history of what this customer has been looking at on the Web site. For example, consider the scenario that Figure 3.1 shows—an insurance service center. What sort of reaction would this company get if, when the customer clicked “help,” a service rep were to come online via IPT linked to a Customer Relationship Management (CRM) front end and speak directly to the customer? “Good morning, my name is Mary. It looks like you’re trying to find information about our new policy options for your automobile coverage. I’m sorry the Web information wasn’t more helpful. How can I help you?”

43

Chapter 3

Web Server IPT Gateway

PSTN

CRM App Server

Internet

Figure 3.1: Customer service and CRM in the integrated IPT environment.

The call center can just as easily support Internet users as PSTN callers. Computer telephony integration at its finest and most developed can provide a powerful competitive edge to customer service organizations. This evolution in the network and change to the corporate culture require both time and focused effort. They evolve continually as new behaviors are learned. Although no company can jump from legacy telephone systems to a fully integrated voice and data network overnight, IPT might provide a point of entry for your company to begin moving into a competitive converged network. Lower Costs As mentioned earlier, lowering cost was seen as an early driver for migration to VoIP. Cost reduction was seen as coming from multiple fronts. First, the consolidation in infrastructure to a singe network was viewed as a source of cost savings. The problem with this logic is that most businesses standardized on a single, high-grade Category 5 or Category 6 twisted pair wiring scheme years ago that can handle the needs of all forms of multimedia information. The old dual-mode cabling of twisted pair for telephones and other types of cable for data workstations has been replaced with a standard, unified wiring scheme. Bottom line: there is not significant cost savings today in the physical plant. A second area of cost savings was the reduction or elimination of telephone costs through the use of the IP network to replace the old phone network. Organizations that started with this idea often find that their existing IP networks are usually not large enough or robust enough to handle the new load of telephony and multimedia demand without being enlarged and made more redundant, and that the cost for VoIP-based outside phone services is about on par with today’s cost of traditional services.

44

Chapter 3 A third area where substantial cost savings were envisioned was personnel costs. The general thinking was that because voice is just another IP application, it could be put on the IP network and managed by the data team so that all the traditional telephony people could be terminated, thereby creating a substantial ongoing cost savings for the organization. This logic is faulty, as organizations that do a wholesale purge of their old telephony departments go without key services. There is just no way to replace the knowledge and expertise. What happens is that organizations often use the migration to a new IPT system as an opportunity to sort out their data and telephony personnel, keeping the best and brightest from each department. What emerges, however, are single super-techs with multimedia capabilities that are well-suited to work in any company and cost more in training, salary, and benefits. It is a whole new era and a proper program must be developed to ensure that the needed skills are available, and this must also include the consideration of service providers and system integrators who will have the hiring headaches and share the cost and availability of their stables of skilled multimedia workers over all clients. The Myth of Bandwidth Savings There was a widespread myth associated with early VoIP services that all you have to do is add VoIP to your network and it will work just fine. Vendors said it will fit in between the unused spaces in your data network. Don’t believe it! Voice service is mission critical to the operation of any business. Anything you add to your network—whether it’s an application, new software, or new devices—will impact performance. If your equipment vendor tells you to add their product and your “voice will ride free,” be very skeptical. You will get what you pay for, and it’s crucial that you protect all your business network services, whether they are voice or data. One early driver behind packet telephony migration was to move voice traffic off the more expensive PSTN onto a cheaper network, like the Internet. This was purely driven as a costreduction measure. Although this method presents an intriguing argument, it also presents a true “apples to oranges” comparison. The PSTN and Internet are regulated, managed, and priced under completely different methods. The PSTN is a highly regulated environment, while the Internet still remains, for the most part, an unregulated data service. One approach to providing quality to a network has been to exponentially increase the carrying capacity or bandwidth of the network. This has historically been referred to as gigabandwidth. With advances in LAN switching technology, Gigabit Ethernet is now fairly common in the backbones of corporate networks. 100Mbps to the desktop is commonplace. Many pundits and technologists have argued that with the exponential increases in optical networking, we have achieved a glut of bandwidth. Some have even proclaimed that “bandwidth is free,” although we all seem to be paying a higher price that we would like to pay. Certainly any IT manager who pays for WAN circuits will tell you bandwidth is not free. Bandwidth is a finite resource in your network. CPU processing cycles are finite too. All your business network services are additive, each consuming available resources. An opposing argument is that providing a user with gigabandwidth is merely a temporary patch or workaround. The Internet has done more to encourage consumption of data than the early designers ever imagined. End users always find creative uses of technology that the inventors or designers never dreamed of. Today, the idea of a gigabit data stream to the desktop seems to eliminate any need for QoS mechanisms.

45

Chapter 3 The question becomes whether some traffic is more critical than other traffic. For many businesses, telephone traffic is the most mission-critical application there is. A call center that takes catalog orders or makes airline reservations cannot function without telephony. Assigning a lower priority to email and Web traffic is a sound business decision for such a company. In order to do that, some prioritization scheme is necessary, whether it’s one reviewed here or some entirely new approach. The best solution to QoS issues involves the well-being of the entire network. Processing power must improve. Bandwidth must increase. Traffic engineering techniques must evolve. Prioritization for mission-critical traffic must be implemented. There is not one single “silver bullet” solution that can placate all the demands placed on IP and the Internet today, because the problem is not one single problem. Only a complex solution will effectively address the myriad performance concerns in today’s networks. Access Charge/Local Local access charges have been challenging for many businesses. Large enterprises might have a distributed PBX architecture with a comprehensive “least cost routing” methodology implemented across the corporation. These local access lines, typically T1, E1, or J1 lines, were at one point, an incredible cost burden. The cost was offset by routing calls inside the private voice network of a large company and dropping them off to the local telephone network as close to the destination as possible. In essence, companies replicated the routing of the PSTN within their own private telephone network, providing the lowest cost access based on their own infrastructure. Although this approach still works in the packet telephony world, the use of the ubiquitous Internet and the lower cost of high-speed packet interfaces often require a rethinking of how calls are routed. Many companies, however, will develop one strategy to get them through the migration period and another strategy for the post-migration period. Any multi-national organization must also consider the legality of packet telephony, or at least a bypass of the national Postal, Telephone and Telegraph (PTT) authority in any country in which they work— as well as the legality of encryption if that technique will be used to keep the packet voice secure. Long Distance Long distance access was reduced for many companies through the local access trunks provided at each remote office. Companies used creativity in trunking PBXs and key systems in smaller offices to make the absolute best use of expensive carrier resources at the cheapest possible rate. Many large enterprise companies built telephone networks that rivaled those of small telephone companies. PBXs were linked or meshed together using point-to-point circuits and proprietary protocols, establishing a telephony network that allowed the customer to create routing priorities. Although this setup has helped reduce the cost of long distance calling in many cases, it has added complexity to the customer voice network and put the onus on the customer for making path determinations about call routing.

46

Chapter 3 Intra-Company

Intra-company voice traffic has been solved as a problem for many years. Companies have established a combination of voice trunks and voice channels on the corporate data network that allow hauling internal telephone calls from coast to coast and around the world. PBX vendors provided IP interface cards to simplify inter-PBX trunking over IP networks, and in many cases, a company simply carved voice trunking capacity out of the data network bandwidth to connect voice systems. This approach was often presented by vendors as “free VoIP” on the data network, however, as Chapter 2 explored, this is a fallacy. More accurately, this was sales and marketing hype that caught the attention of some customers. What companies did find is a financial savings in using the same carrier network infrastructure to transport both voice and data traffic. Nothing rides free, but the savings of a single infrastructure and single billing system has proven quite significant. Extra Company

Telephone calls to the outside world are the lifeblood of many businesses. We call customers, vendors, and business partners every day. As companies scrutinize expenses for inbound and outbound telephone calls, close financial management motivates business managers to find creative new ways to reduces costs. Support Costs In addition to the engineering resources addressed earlier, many organizations look to cost savings in other support areas. Just as in the network engineering area, don’t divest yourself of other critical support resources such as the service desk and the technicians who handle moves/adds/changes. Instead, team your voice and data specialists together as a new unified communications team, extracting the valuable knowledge from every resource at your disposal. Don’t let the knowledge of your services, your network, and your requirements slip out the door through some misguided effort to reduce costs. The cost reductions must be won over time. They aren’t necessarily real for every business and they don’t come immediately. Help Desk/Support

Help desk support for telephones is generally simple. Everyone knows how to use a telephone. Many companies conducted user training at the time a new PBX was implemented and handed out basic feature guides from their equipment vendor that explained how to use voicemail systems and the various features of the system. In the integrated environment of VoIP/IPT, these features may be delivered to telephone sets that can be treated like traditional telephones. More often, integrated system introduce some element of data workstation feature management. For example, users can often manage their own telephone set button configuration and speed dial lists from a Web interface via their workstations. It’s critical to remember that as the services converge onto a single infrastructure, your corporate Help desk will be getting calls about new services and question about things they too will be unfamiliar with. It’s vital that the Help desk and support teams be involved in the migration process and participate in early training efforts it order to effectively support ongoing daily business operations.

47

Chapter 3 Moves/Adds/Changes

When implementing an integrated services project, moves/adds/changes must be taken into consideration. This is hugely important in a traditional telephone environment but even in an IP telephony environment there are still many potential problems. Employees often move his or her workstation, which retrieves an address from the network using Dynamic Host Configuration Protocol (DHCP). Telephones, on the other hand, often had to be moved by the telephony administrator, reprogramming the PBX to deliver extension number and features to a new physical wire in the new work area. With integrated services, this process may be simplified but it cannot be overlooked. For instance, many systems allow the users to make all needed changes but often the users lack the technical sophistication to make the needed changes and even the most savvy users are rarely given access to firewall and security systems that often stand in the way of completely transparent moves adds and changes. In many cases, the $12 per hour telephone technician—whose primary skills involved being able to read a floor plan, locate wire pairs, and use a punch-down tool—is being replaced with an $80-to-$120K per year IP engineer with knowledge of fire walls, IP router configuration, subnet address masks, and an entirely new set of skills possessed by a limited number of workers. Remote/Nomadic/Mobile Workers Integrating voice services with VoIP solutions provides an excellent tool for augmenting remote worker resources. Your call centers can now be staffed with people working from anywhere. For some call centers, this provides an opportunity to hire new, less costly staff that would have previously been unreachable. For example, stay at home mothers can now provide telephone support with full access to corporate resources. The widespread availability of broadband Internet, coupled with VPN technologies, provides a data linkage virtually anywhere. VoIP and IPT solutions do the same thing for voice services. Enabling New Strategic Applications Strategic new business applications are many and varied. Your company might have industryspecific applications that don’t apply to other industries. Inventory and supply chain management, for example, is crucial in the manufacturing sector but plays a much smaller role in the financial services sector. Business drivers, market shifts, and process re-engineering all play a role in determining what strategic new applications a business might deploy. CRM systems are permeating many new businesses. Enterprise resource planning (ERP) solutions are another factor for many organizations. It doesn’t matter whether you’re deploying Oracle, SAP, or a Web application; the odds are high that any strategic change in your data services applications will ripple into telephony services in some way. In many cases, integrating telephony services will be a key business change enabler in and of itself.

48

Chapter 3

Worker Productivity Worker productivity will go up. Worker productivity will go down. There will be iterative cycles of improvement. Although your staff may be provided with new tools and new options that can enable them to work more effectively, don’t overlook the learning curve. Most people learned basic telephone usage at a young age and have spent most of their lives dealing with telephones. Shifting that process to a softphone running on a PC, connected to a wireless network, integrated with a CRM application presents a new way of working. Integrating unified communications into the corporate environment introduces the potential for a large cultural change. It’s absolutely vital that workers be involved in understanding the changes and how their work will be impacted each step along the way. The more engaged workers are, they more quickly they will find ways to use the new environment to increase productivity. Most workers are driven to get things done, but they also need to understand what the tools they have to work with can do—and how those tools work. Road warriors can use these technologies to enhance accessibility. Traveling executives can use their office number remotely from a hotel room. Engineers can still work while in training or away on a project. Integrated voice services provide the opportunity for not only enabling this work flow but doing it all transparently. A customer calling a salesperson doesn’t need to know that the sales rep is halfway across the country in a hotel. The office telephone number can become truly portable. And the coming advancements in fixed mobile convergence and IP multimedia subsystems (IMS) will couple service more tightly to the remote worker than was ever before possible. When evaluating systems, it’s easy to fall into the “feature/function trap.” Vendor selections are often made based on features and functions, with cost comparisons driving final decisions. For your workers, it isn’t just about the features that exist, but how they are used. A worker who is used to having a 32-button telephone set that shows eight coworkers’ phone lines with status indicators will have to change work habits to effectively use a softphone. Engage workers throughout the entire migration process from planning to production to maximize their productivity on day 1, day 10, and day 100 after deployment. Obsolescence of Older System Older systems become obsolete. Traditional Time Division Multiplexed (TDM) PBX systems once had a practical lifetime of 10 years. Many outlived that, but many were also obsoleted early because of technological advances. Data networks have tended to be re-architected in a shorter life cycle. Data networks are often redesigned every 3 to 5 years. Switching and routing technologies have advanced quickly, making complete redesign of data networks cost effective in a shorter life-span than older PBXs. Consider not just the obsolescence of a legacy telephone system but also the data network life cycle when developing migration strategies to move to VoIP or IPT solutions.

49

Chapter 3

No Longer Want to Invest in Old Technology Obsolescence is one driver, but for many companies, especially market leaders, there are compelling business reasons to migrate to newer technology. It’s not uncommon for an enterprise to move to new technology, not when older systems are at end of life, but when the end is visible on the horizon. Manufacturer Discontinuation of Product and/or Support Manufacturers discontinue support for older products every day. In some cases, the parts become scarce, making hardware repairs a challenge. In many cases, manufacturers will consciously choose to end the life of a product in an attempt to encourage customers to move to newer systems. In the history of business telephony, this was often called a “forklift upgrade.” The analogy being that a forklift was needed to remove the old equipment to make room for a complete new system. If your equipment manufacturer has put your company in a “forklift upgrade” situation, it may be important to consider how firm your relationship is with the manufacturer. Vendors play a dangerous game with this strategy, as customers are presented a perfect opportunity to take a system design back out to competition on the open market. This technique has backfired on many vendors, costing them key customers. Increased Support and Maintenance Costs As systems age, they become more expensive to support and maintain. Parts may be less available for hardware replacement. Technical skills to support older systems will naturally decline as these systems phase out of useful life. If you continue to use a 10-year-old system when much of the market has moved to other technology, the support costs to maintain an effective staff can easily rise. What it costs to support and maintain a telephony system today may be 30 percent more than it was when that system was a widely deployed standard implementation. Another cost factor might simply be technology related. Newer hardware and software often introduce new patching and upgrade methodologies that are more cost effective than techniques used in the past. That can drive down support/maintenance costs for new systems in comparison. Make sure you review all the data and factor in the ongoing overhead in dollars, time, and skill set to support each of the options you consider.

50

Chapter 3

Lack of Expansion Capability of Old System Hardware and software solutions have practical limitations. Telephony systems are introduced to the market at a price point to ensure widespread adoption. As time passes, technology advances, and businesses advance too. A key system designed to support a 10-person office 5 years ago may well be stretched to capacity today, as that office may have 30 people. Hardware could be utilized to the max already. Expansion capability isn’t just hardware. Older hardware solutions might not be able to support newer features your business requires. Expansion in software can only be supported to the capability of the hardware. In your evaluation process, you can’t just look at the requirements of today. The investment of technology a business makes is a strategic investment. It has to support the operational needs to the moment, today, as well as growth and evolution to support the strategic needs of the business for some foreseeable future. For some companies, that might be 7 to 8 years, and for others it might only be 3 to 4 years. Incorporate your strategic direction plans for the business with your strategic plans for technology. The technology won’t do you any good if you don’t build it to support your business.

Setting Proper Expectations The bottom line for the success of any project is setting proper expectations. The executive management team may have very clearly defined expectations based on ROI and the size of the investment. Users may have an entirely different set of expectations based on workflows and operations processes. If you build your entire ROI case around the technology but implement an integrated service in a way that increases the labor effort of your staff, the overall result may be counterproductive. Everyone needs to know what to expect and what is expected of them. The only way to really be successful in any large project is to clearly define and communicate expectations. Like the network, the telephony system, and your business as a whole, the life cycle of an implementation project is a repeating pattern of setting and reviewing expectations. Management Expectations The executive management team focuses on key strategic factors. Capital expenditures (CAPEX) address the investment, and are often tied to ROI. Operational expenditures (OPEX) take into account the cost of operations and support; they’re tied more to the total cost of ownership (TCO). Technology can often drive process re-engineering that changes workflows. Technology is often molded to fit business needs, but just as often, business processes are modified to leverage some feature of the chosen technology. You must also consider the return on effort and factor the work effort involved into the total cost of implementation and ownership. Implementing VoIP or IPT comes with a broad general set of management expectations that vary widely, yet are all accurate. One view is that IPT will save money. This view generally assumes a modest CAPEX spread across all users of the service. Using cost recovery as a lens through which to view CAPEX helps balance the total CAPEX against the true investment and determine how quickly a solution might pay for itself. The money saving view often sees that OPEX can be reduced by saving on recurring cost. For many companies, this is based on a shift from a provider-managed service such as Centrex to a self-managed-IPT service.

51

Chapter 3 Another view is that VoIP may cost more. One migration can lead to another. Some VoIP deployments have been plagued with network upgrade upon upgrade, but this is often because a comprehensive readiness assessment wasn’t completed on the front end. Many business managers find that in their environment, CAPEX is as high as it would be to continue using traditional TDM-based PBXs. OPEX can be variable and particularly difficult to measure when you broaden the scope of the IT staff’s role. The impact of requiring a new key competency or technical skill in a networking staff can inflate OPEX for the first year or two as the staff climbs up the learning curve. There still may be tremendous value in integrating strategic business or multi-media applications. A third view is that it doesn’t matter. It could be a cost-neutral decision either way. This is why some companies take cost reduction off the table in their strategic planning of technology. It’s important to know your core competency and focus on that. The business you’re in is the business you have to support. The technology choice needs to be driven to support the strategic direction of the business. Don’t let the technology drive your business. Leverage the technology to strengthen your business. Integrating voice and data is a complex project, filled with potential pitfalls. Comprehensive planning, network readiness assessment, and project oversight is absolutely vital to the success of an IPT implementation. Every step should be documented. Every test plan should be thoroughly reviewed by multiple team members. As you begin the implementation, you’ll face a series of potentially service impacting events. Circuits might be upgraded. Routers and switches might be replaced, and firewalls might be reconfigured. At every step along the way, thoroughly test the new environment. And for each of these steps, document a fallback plan to ensure business operations aren’t disrupted if unexpected problems arise. Build a comprehensive plan for success with full management buy-in. End-User Expectations It’s essential that end users be included in the process from planning to implementation. They are invested in the success of unifying communications services. They need to know that although things will be different, different isn’t bad. During needs analysis, you’ll need to understand their requirements. As the project progresses, setting expectations about how things work, voice call quality, and new features are important to them. Collaborating with users makes them part of the process and part of the project and gives them an investment in the success. Treat them as advisors and partners whose business needs you’re supporting. Training is often available through the vendors whose solutions you’ll buy. Take full advantage of vendor training opportunities. Train the trainer sessions with key users is an approach that’s proven very effective. Even the final cutover implementation needs to actively engage end users. They will be your primary system testers. Use them to help develop test plans, and they’ll feel their needs are being addressed. Be proactive and responsive. Another very effective technique is to activate a special cutover Help desk for several days before and after the “go live” date for your new system. This approach builds confidence and makes for satisfied users. When the users are satisfied that they’re getting what they need and understand how to use the system, your project is well on its way to being successful.

52

Chapter 3

Inventory of Existing Capabilities As you begin the assessment part of the project, you must perform an audit of the capabilities of your current telephony system. Such an audit provides key insight into potential problem areas. What you know about your operating environment provides the foundation on which you build your assumptions. The more you know, the more accurate your assumptions will be. There are some specific areas to audit that help increase your chances of success: •

•

•

•

•

Network infrastructure • Identify circuits used for wide area networking and Internet connectivity • Inventory the network hardware—routers, switches, cabling. • Document software such as router OS versions; this is a good time to get your routed network all on a common foundation Current business data applications • What servers are in place today? • What is changing or evolving in parallel? • Identify any linkages between your existing business applications and voice services Security • Inventory/audit your security environment • Ensure that firewalls and other security systems will allow VoIP traffic • Determine capabilities for moves/adds/changes allowed by your security system • Assess legal issues regarding security and encryption for the areas in which you will be operating Management tools • What network management tools are being used in your data center? • What server/application management tools are you using in your data center? Current voice services and equipment • List all PBXs and key systems in your organization. For each, include its “End Of Life” date and/or date at which it will be fully depreciated • Document any inbound and outbound call center environments • Document expected telephony performance • Inventory all existing telephony features, determine their importance, and decide whether they will be available in the new telephony environment; this audit is covered in the next section • Measure your current busy hour calling patterns • Intra-office • Intra-company (between offices) • Offnet for each office (incoming and outgoing calls [local, long distance, international]—not to other offices) • Fax machines • PSTN channels/lines for each office

53

Chapter 3 Existing Telephony Performance Expectations Audit Ideally, you should quantify TDM service quality aspects that you want to carry over into IPT, either to ensure you build a system that meets user expectations or so that you have defined criteria to which to hold your systems integrator accountable. Quantify and define requirements around: •

What is a reasonable length of time to wait for dial tone? At what point do you start jiggling the hookflash or start punching in the number anyway?

•

Once you’ve finished dialing a number, how long do you wait for a ringback tone before hanging up and trying again?

•

For every 100 calls you make where you correctly dial the number, how many of the 100 do you expect to succeed?

Based on responses to these questions, define a set of SLA requirements against which the IP telephony environment will be measured, to evaluate its success and ensure that it meets customer expectations. These SLAs should include: •

Availability expectations—This is usually expressed in uptime percentage of the telephony service from an end-user perspective.

•

Throughput—Stated as successful call completions as a percentage of call attempts. You might want to provide different throughput requirements for different call types. For example, you might require a 99.9% successful call completion rate for all incoming calls to a call center, while requiring that only 97% of all internal calls complete successfully.

•

Call setup and teardown boundaries; for example: o 95% of all call attempts get dial-tone within 2 seconds while the remaining 5% get dial-tone within 4 seconds o 90% of all call attempts get ringback tone within 6 seconds while the remaining 10% get ringback tone or congestion tone within 9 seconds

•

Voice quality expectations, usually expressed in Mean Opinion Score (MOS) bands of 0.2 range—For example: o 85% of calls must have a MOS score of 4.2 or higher o 95% of calls must have a MOS score of 4.0 or higher o 4% of calls must have a MOS score of 3.8 or higher You might also want to consider defining different voice quality requirements for interoffice (that is, calls that cross the WAN) than you do for calls within a local office.

54

Chapter 3

Existing Telephony Budgeting, Relationships and Tools Audit Also worth considering as part of planning your IPT rollout are the potential changes in responsibility for paying phone bills and impacts on business processes: •

Prior to IPT, each region/office may have had their own interconnect and telephony budget.

•

After IPT rollout, who will pay the telephone bills? Same as before? Or maybe the NOC team charges each department for their telephony service and holds the entire budget for telephone bills?

Consider existing service provider relationships—telephony, network, and desktop management. Also consider existing management tools—event manager, network management, and server/application management tools. Which (if any) of these existing relationships and tools should be integrated into your new converged environment? Existing Capabilities Audit A logical approach to auditing the capabilities of the existing system would seem to be to go to the documentation for the existing system, check to see which features have been paid for and/or implemented, and use that as a guideline for the new system. This approach only gets you part of the way there. What is really needed, beyond knowing what features are available, is knowing what features are actually used. This second requirement needs a great deal more research but can help you ensure that your new IPT system will provide every feature your organization needs, while avoiding the unnecessary effort, complexity, and cost of acquiring an IPT system that provides features that your organization simply does not need. There are two general approaches to determining which phone features are used and which need to be replicated in any new system. One is network-centric and one is user-centric. The networkcentric approach involves gathering network traffic, such as signaling traffic, to ascertain which features are being used. Although this approach makes perfect sense to an engineer, it often neglects features that are local to specific phone sets. Another network-centric approach is to check the PBX or Centrex configuration information for the features that are available for each line and to use those as a basis for the new system design. Although this option comes closer to understanding the user’s needs, it is still, at best, an indirect method. The best way, actually, is to ask the users! The users will be able to explain not only the features that they use but also the relative value of each. One must also consider that certain phone features that may be unimportant and replaceable in one industry are critical to others. “Wake-Up Call,” for instance, is a feature absent from most business telephony systems but one that is critical for the hospitality industry. “Skills Based Routing,” the ability to route a call based on the skills of a particular individual call taker or class of call-taker, might not be important in a hospital environment but may be of the utmost importance in an insurance company. These are some of the reasons to ask the users rather than make assumptions. The user-centric approach begins with the network-centric approach, but instead of jumping right to the inventory and analysis, the list of features is translated into a survey tool to be used by the phone user to indicate features that they employ. Figure 3.2 shows a list of many of the most common features. 55

Chapter 3

Figure 3.2: Traditional telephony features.

Once a comprehensive list of potential features is developed, a suitable user survey approach must be created. Once again, the obvious approach is not the most suitable. The obvious approach is to provide a list of all possible features and ask the user, at the very least, to check off which features they use and, at the very best, to rate the features in terms of their importance on a 1-to-3 or 1-to-5 scale. The problem with this approach is that the users often don’t understand the technical name for a feature. The obvious solution is to provide descriptions of the features, but that still leaves the problem that not all users use all features all the time and that they might leave off an important feature.

56

Chapter 3 Another approach is to provide some type of card with the features listed so that when a user uses a particular feature, they can simply “tick off” the feature. Although this approach works well and provides a more-or-less accurate audit over a period of time, it generates a lot of cards. The total number of cards can be reduced by having the user check off the feature the first time they use it and use hash marks to indicate how many times they use the feature during the study period. This reduces the number of cards, but analysis is still problematic. A further simplification can be achieved by choosing representative users for each department, job function, or operational area. Doing so assumes that each executive secretary will use the same set of functions and features—not always an accurate assumption—and that each call center operator will use the same set of features and functions—more likely to be true within a job category. Entry-level operators will likely use the same subset of functions; supervisors will likely use the same set, and so on. Once a cross-section of users is identified, one more simplification step can be achieved. Through the use of a specially designed survey card, which is drilled at specific points around the periphery to align with audit questions, it is possible to greatly ease the survey and audit process and quickly analyze results. The creation and use of cards of the kind shown in Figure 3.3 can greatly enhance and speed information collection and analysis.

Figure 3.3: Sample telephony capabilities audit card.

57

Chapter 3 It is also not required to print capabilities on a card that will not be used by a specific audit group. “Wake-Up Call,” for instance, is not required on a call center operator’s list of possible features. Different colored cards should also be used for each functional area to allow common and unique requirements to be identified. Synchronizing Existing and New Capabilities Once the cards have been marked by having the appropriate holes removed, they can be stacked and jogged to align the holes; then information can be derived from the card deck regarding features of importance. This is done by inserting a long stick, such as a small diameter chop stick or knitting needle, into a hole and then shaking the stack gently until all cards with the same hole punched out drop from the deck. This process will result in a list of features that are used in the current system and that therefore must be supported by the new system. Users who only have requirements that are supported by the new system may be placed into the first group for migration. A list of those requirements will also need to be assembled. This process will also result in a list of features that were not known to be requirements of the new system but which must be included in the RFP. Features not supported in the initial release of a new system but which will be supported in subsequent releases can result in users needing those features to be placed in a second group for migration. This process will also result in a list of features that are never intended to be supported by the new system. The needs of these users must be somehow synchronized with the capabilities of any new packet telephony system through retraining, reworking of the need for the feature, or a rethinking of a move to the new packet telephony system until the features are available. In many cases, it should be possible to extend the lifeline of an existing system through the use of a feature or gateway or other mechanism to allow it to continue to provide the needed feature for the foreseeable future. Voice over Packet The broad term VoIP has been accepted by the industry and consumers to refer to a new, inexpensive way of making telephone calls using the “Internet.” Although “insiders” understand that the IP network is not always the Internet, the common definition is convenient and easy. What VoIP actually means is that the voice information is sent using proprietary signaling protocols, such as Cisco’s Skinny or H.323 and increasingly using Session Initiation Protocol (SIP). The term VoP, or Voice over Packet, is inclusive of VoIP and is a more general term meaning the coding of voice into binary and transmitting over a broader range of options, including ATM, Frame Relay, and DSL. By considering a wider range of voice over packet options, not just VoIP, an organization can often maximize use of their current networks, avoid dealing with regulatory issues in certain countries, and extend the life of existing assets.

58

Chapter 3

Network Readiness Assessment The chapter introduction talked about a race car analogy. At this point in the chapter, we have talked about the reasons to join the race and how to properly assess that the new race cars being purchased or built, at the very least, have all the capabilities of the race cars being replaced and, preferably, are more powerful, faster, and have more features. We now turn our attention to the race track: the existing IP network. Once the decision has been made to move to VoIP or IPT, an extremely important part of the planning process is to evaluate not just whether the existing IP network can handle the predicted call volumes while meeting the exacting real-time performance criteria associated with telephony, but also what enhancements are required to the IP network to ensure that it can do so. More than 60% of organizations that have already rolled out IPT did require some level of IP network upgrade and, in a not insignificant number of cases, the cost of the network upgrade ended up being equivalent to or greater than the cost of the entire new IPT system. Many existing networks today evolved from departmental local area networks (LANs) that have grown over time. In some cases, regular, methodical redesign work has been performed every few years. Just like the PSTN undergoes constant traffic re-engineering changes, some IP networks have been finely tuned and redesigned to support new data applications. For others, the applications in use today may be very different than the original applications were. It’s important to fully understand the network performance requirements of VoIP services being added and how they will impact existing network services. Many companies begin voice integration projects fully understanding that network upgrades will be required. Network nodes such as routers, switches, and firewalls may already be running at high CPU utilization. They might not be able to support VoIP services without upgrade or replacement. Existing wide area network (WAN) circuits may already be overburdened carrying the existing data applications. Network bandwidth, robustness to single points of failure, and processing capacity are key factors in deploying any new service like integrated voice. One of the most common mistakes, and one that has doomed more than a few IPT projects to certain failure, is skipping the network readiness assessment and jumping directly to implementation. There are four primary reasons for skipping the network readiness assessment, and all appear to be viable reasons until the project completely falls apart leaving the IT/telephony group with no credibility and the organization with no telephony system. The typical reasons are: •

“A network readiness assessment takes time and money. VoIP/IPT is just another IP application and we know it will work. Why waste time and money proving what we already know?”

•

“We hooked up a VoIP server and some phones over the weekend and everything worked beautifully! We can now implement with confidence.”

•

“We have enough bandwidth in our network to transmit the entire World Book Encyclopedia twice every second. If our network can do that, we can definitely handle VoIP and, besides, our current network utilization is below 30%.”

•

“We can’t upgrade our old TDM system anyway: we have to go to VoIP. Why do an assessment? We’ve got to move to VoIP anyway!” 59

Chapter 3 The following horror stories, from real organizations who did not do network readiness assessments, will illustrate the inaccuracy, and risk, associated with not performing a comprehensive network readiness assessment before moving the first live call to the existing IP network. Horror Story #1 A large US retail chain went straight to pilot without having completed a network assessment. Their call quality was terrible, the call setup delays were frequently more than 5 seconds, and down times were frequent and protracted. The pilot was deemed a failure, the roadmap to IP telephony was discarded, a lot of money was wasted, and they are still using their traditional TDM system. Lesson Learned: Network readiness assessment is a critical part of any move to VoIP/IPT.

Horror Story #2 A large government agency went straight to live implementation without either assessing their network or executing a pilot. Telephony traffic saturated their network, with the result that their call quality was consistently unacceptable and all other distributed applications in their network experienced long delays resulting in extreme telephony and data user dissatisfaction and low productivity. To resolve the problems, the customer had to back out 75% of their IP phones. They are now in the process of doing their network assessment on a live network and will have to try to redesign and upgrade their network without impacting their live users. Their original ROI forecast has been blown away and their current budget analysis is that their IPT rollout will end up costing them more than double their original forecast. Lesson Learned: Any change to the existing network has far-reaching implications on overall organizational operations and productivity. An investment in network readiness assessment ensures accurate planning and budgeting, minimum impact on the organization, a dramatically increase in the chances of success, and provides a safe fallback position in case things don’t go as planned.

So, our recommendation? Do an IP network readiness assessment—the effort required will definitely be worth it, as it will help you plan a successful IPT implementation project that: •

Meets its target dates, as both the resource effort and time needed to upgrade the IP network are factored into the IPT rollout plan

•

Comes in on or under budget, as the cost of the IP network upgrade will be factored into the overall cost of the deployment and budgeted for accordingly

•

Meets user expectations of telephony performance without impacting on organizational productivity

60

Chapter 3

The Network Readiness Assessment Process Conducting a successful and valuable network readiness assessment is not as simple as simulating some calls over your IP network and noting the results. Here are some more horror stories from real organizations, but this time, they are from organizations who did conduct a network assessment but who still ended up with serious problems after going live with IPT. Horror Story #3 A medium-sized enterprise set up their automated assessment tools to run overnight so that the telephony traffic wouldn’t degrade the network QoS demanded by their users. The results of the assessment showed acceptable voice quality for all calls. An initial implementation was then rolled out and the voice quality was very low because the voice traffic was now competing with other IP applications for bandwidth. Lesson Learned: A theoretical network readiness assessment can be accomplished completely offline to avoid negative impact during initial phases but real traffic must be simulated during real business hours to determine the true impact of the new IPT traffic prior to actual implementation.

Horror Story #4 A large distributed IT enterprise ran their own assessment and completed their network upgrade based on this information. However, when they completed implementation, they found that a large percentage of their off-net call attempts were suffering from very low voice quality. After acquiring a management tool, they used it to identify that, during peak hours of the week, there was insufficient bandwidth available to their primary PSTN gateway. It turned out that they had forgotten to simulate off-net traffic in their assessment. Lesson Learned: Use existing call records from telephone carriers, PBXs, and other sources to ensure that the network readiness assessment includes all calls, both on-net and off-net, and that call volumes can be supported during the busy hours.

Horror Story #5 A university was conducting an assessment and based the distribution of their simulators on their network diagrams. One campus was shown, in the plans, to have a 10Mb link, so they designed the assessment to have 80 calls occurring simultaneously across this link. They found that the voice quality for these calls was appalling. They lowered the number of calls being simulated and discovered that acceptable call quality could only be achieved if the number of calls across this link was limited to 15. This made no sense, so they sent an engineer out to investigate. It turned out they did have a 10Mb switch on that link, but only 1 E1 card (that is, a 2Mb link) was connected to it—their network diagrams had been incorrect. Lesson Learned: Always validate configurations, system capability, equipment inventory, call capacity, and so on.

61

Chapter 3

In a Perfect World In a perfect world, with an unlimited budget of time and money, the network readiness assessment would be a carefully applied four-step process. The first step would be to gather information from the live network. This information would be merged with projected network traffic loads anticipated from new IP telephony applications, which would then be subjected to static modeling, which is the process of creating what would, in effect, provide a theoretical snapshot at a moment in time. The output of the static model would provide the input for dynamic computer simulation, which would provide the basis for a proof of concept test. Then, and only then, would an actual field test and field trial be undertaken. Most organizations know that the investment of time and money in such an exhaustive series of steps is not warranted. That is, of course, until they either fail in their attempts to migrate voice over to their IP network or come very close to such a failure. It is only then that they appreciate what they have risked. In the Real World Sadly, in the less-than-perfect world of time and budget constraints where most of us live, it is generally not possible to go through this entire network readiness assessment process. Therefore, the minimum set of steps recommended to get the best value out of your assessment is to estimate projected traffic, perform a simple overnight assessment test run and, if successful, move to a daytime assessment test run, starting with low assessment traffic levels and gradually building up to simulating the estimated traffic. If this set of procedures is successful you could then move to a pilot phase. However, by adopting this minimal approach, you still have the opportunity to stop simulating traffic if the test begins to seriously impact the network and other applications and users. Unfortunately, many organizations move straight to a pilot, usually with less than acceptable results. Understanding the Key Metrics of a Network Assessment Whether you are conducting your IP network readiness assessment yourself or using an assessment service provider, it is good to understand the key metrics and factors involved. The PSTN has evolved over time into a finely tuned network that has been optimized for delivering voice traffic. In business especially, for years you heard the phrase “toll quality voice” to describe a service level suitable for business. And of course, you have heard a description of service good enough to hear a pin drop. The Mean Opinion Score (MOS) described in Chapter 2 has been the accepted benchmark for measuring this quality. This network has been built and conditioned to do one thing really well—deliver voice conversations. Your data network has been optimized and honed to support the unique set of data applications your business requires. Integrating telephony services will place new, sometimes-inflexible demands on the resources of your network. Data networks use IP to deliver traffic. IP is described as being unreliable and having no guarantees. It is a “best efforts” protocol. It was designed to carry sporadic, unpredictable traffic loads that burst to peak volumes at times. You’ve probably heard the phrase “bursty in nature” to describe most IP traffic. At the transport layer, above IP, there is Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). TCP provides the ability to guarantee delivery through a process of synchronization and acknowledgement messages, but this delivery guarantee also adds overhead. 62

Chapter 3

Voice Traffic

IP Traffic

Connection-oriented—A dedicated path is established for each telephone call.

Connectionless—Voice conversations are digitized and bundled inside IP packets, then transmitted over the best route based on routing protocols.

Calls are long in duration (3 minutes on average).

Although IP telephone calls have the same duration as traditional ones, the individual packets are small, so conversations are cut up into many packets, typically lasting only a few milliseconds.

Delivery is guaranteed once the call path is established.

Best efforts are made to deliver traffic but there are no guarantees.

Designed to use specific bandwidth. The PSTN uses a 64Kbps voice channel.

Uses the bandwidth that is available.

Real-time voice traffic is very sensitive to delay.

IP data traffic is often delay-insensitive.

Table 3.1: Voice and data traffic comparison.

What you see is that IP data networks have very different basic characteristics than traditional voice telephony networks: •

Unlike the dedicated circuit connection the PSTN establishes to carry a phone call, IP network traffic is composed of packets. Packets tend to be short in duration, although a phone call consists of hundreds, even thousands, of packets. Packet data is generally bursty in nature. Since packets are small in size, they are of short duration in terms of time spent on the network, and they can be routed over many diverse paths and carry addressing or routing information inside each packet. IP doesn’t establish circuits, it’s connectionless.

•

IP packets aren’t generally delay-sensitive. Email messages can be delayed 60 seconds, or 60 minutes without adverse impact to the “conversation.” IP traffic is historically nonreal-time traffic. Delays are expected in an IP network. IP is a best efforts protocol with no guarantee of delivery. The PSTN is designed to minimize delay and provide nearinstantaneous delivery of voice conversations.

•

IP uses all the available bandwidth when it has data to deliver. IP doesn’t require dedicated bandwidth.

Today, IP networks are designed, redesigned, and constantly reconfigured to carry any kind of traffic. With digitization, any media (voice, video, data files, and so on) can be carried inside IP packets. The most common are data, voice, and video. The challenge you face in unifying communications is in delivering a business-quality telephony service with all the characteristics users expect, such as clarity, fidelity, and near-instantaneous delivery.

63

Chapter 3 To understand the key metrics that will impact on the quality of your IP telephony service, let’s take another look at the results of the Telephony Performance Expectations audit that was described earlier in this chapter. User expectations regarding telephony performance are generally quite consistent, as TDM has trained users to have high expectations. These expectations can be summarized as: •

High availability: A user expects to get a dial-tone every time they lift a phone handset.

•

Call quality: A user expects that, if they dial the number correctly, then they will get either a ringback tone or a busy tone within no more than a few seconds—users become impatient quickly if they get no tone at all and get irritable if they consistently get a network congestion tone or announcement.

•

Voice quality: A user expects to be able to hear the other party in the call clearly and does not want to hear echoes of their own voice. A user considers it to be unacceptable if they have to abandon the call because of poor voice quality.

It’s important to note here that call quality and voice quality are both critical, but very different aspects when you look at quality of service in VoIP solutions. Voice quality focuses on the fidelity and user experience from the sound of the call. Call quality includes voice quality as a factor, but also reaches more broadly into the total user experience. How quickly was dial tone provided? Did touch tone dialing work on the first attempt? Do connections across the networks, between offices (or hops across the IP network) process quickly? Do calls complete as expected or do they fail after all digits have been dialed? Call setup delays or failures negatively impact the user experience, reducing call quality regardless of the actually quality of the voice traffic.

When a user perceives unacceptable call or voice quality, that user is likely to terminate the call. If this happens frequently, dissatisfaction with the service lingers. This perception is very difficult to overcome, so you are better off to plan and re-architect your IP network in advance, to make sure you don’t end up in this situation. High Availability It’s important to embrace the telephone network services as a mission-critical component of daily business operations. The PSTN has been engineered to provide 99.999 percent uptime of the telephony service, often referred to as “five nines” reliability. This is equivalent to approximately 5 minutes of downtime per year. Corporate data networks have rarely offered this level of reliability in the past. Think about your corporate network. Have you experienced more than 5 minutes of network downtime in the past year? At this phase of the VoIP/IPT project, it’s crucial to understand that some redesign effort may be necessary to provide suitable network reliability to meet the voice and call quality requirements associated with a telephony service.

64

Chapter 3 Five nines reliability is not just about uptime of routers, switches, interfaces and links in the network—it’s about ensuring that the network can still provide telephony service even when there has been an outage of any kind, anywhere in the network. The general requirement on an IP network to deliver five nines reliability is that it can still provide full service to all users under any single point of failure. This requires that there be redundant paths and fault-tolerant or high availability equipment to guarantee availability in the event of the failure of a given node or device. It also requires that, even under any single point of failure, there must be sufficient bandwidth available for use by the IP telephony service so that it can still meet user expectations. In enhancing your IP network to support your IP telephony service, it will be crucial to identify the IP network availability objective that will be committed to the IP telephony application in an SLA. Call and Voice Quality Both call and voice quality require adequate bandwidth, low latency, low jitter and low packet loss. Bandwidth There must always be enough bandwidth in the network to handle both your IP telephony traffic and all the traffic associated with your other distributed data applications, even under any single point of IP network failure. You probably already have a good idea of the usual profile of the bandwidth usage of your existing data applications, but you also need to calculate the bandwidth requirements for IP telephony. To do this, it’s a good idea to first look at your existing TDM call traffic patterns, as the telephony habits of your staff are very likely to remain consistent after you’ve deployed IP telephony. More detailed guidelines for gathering these call traffic patterns are covered later in this chapter, but the basic idea is to base your bandwidth requirement calculations for each office around the number of calls placed and received during the busiest hour of the busiest day of the year for that office. Once you have an idea of the maximum number of simultaneous calls that will be handled by each office in your organization, you’ll next need figure out what your policies are going to be around codec utilization, as this will impact directly on the bandwidth that will be used by these calls. Choosing the Right CODEC

To transmit voice over a data network, the voice conversation must be packetized. To transmit the conversation in packets, you need to digitize it. Voice digitization is not a new technology. In the 1960’s, the US phone companies began using a digital carrier system called T-carrier while the rest of the world began using a slightly different system called E-1. Initially T-1s and E-1s were used as a trunking technology between phone company central offices. Today, the T-1 or E-1 circuit is a common business connection.

65

Chapter 3 Earlier, we saw that voice represents an analog signal. To use IP, you must first convert it into digital format. This conversion is commonly accomplished in the traditional voice services through a technique known as pulse code modulation (PCM), using a coder and decoder (or codec). Using PCM, analog voice conversations are sampled 8000 times per second. A technique called pulse amplitude modulation (PAM) is used to convert each sample into one of 255 possible 8-bit words. Through a process of compressing and expanding (called companding), noise is reduced to help eliminate background hiss and changes in volume. There are several voice processing standards that were developed by the International Telecommunication Union (ITU, formerly the CCITT) to standardize encoding of digital speech. In the PSTN, the G.711 standard is used worldwide. More information about encoding standards is available at the ITU Web site at http://www.itu.int/. Voice Compression Techniques

Different encoding mechanisms use different levels of compression. Although G.711 encoding is widely used, it typically generates a 64Kbps voice stream in each direction (that is, 128Kbps in total for a G.711 call). For some businesses, quality considerations might make a different encoding scheme more suitable. Greater compression of the voice reduces the bandwidth requirement, but voice quality may suffer as a result. Table 3.2 provides a comparison of different encoding schemes that are all widely used. It lists the Codec types and algorithms used, the bit rate and sample size, and the algorithmic encoding delay. It also shows the maximum possible Mean Opinion Score (MOS) for these codecs. It is important to remember that factors such as encoding delays are cumulative with other delay in the network. If the total delay exceeds 250 milliseconds (ms), voice quality will suffer audibly. These algorithms are described in depth in ITU-T standards (http://www.itu.int/) and a wide variety of papers. ITU Codec

Coding Scheme

Bit Rate Each Way

Sample Size

Encoding Delay

MOS

G.711

PCM

64Kbps

8 bits

90%/99.0

40-80ms (reg) 100-250ms (global)

1-20ms

Data

99.7/99.99%

100%/99.0

50-100ms (reg) 150-350ms (global)

1-20ms

Video

99.7/99.99%

>95%/99.0

40-80ms (reg) 100-250ms (global)

1-20ms

Audio

99.5/99.97%

>97%/99.5

50-100ms (reg) 150-350ms (global)

1-20ms

Data

99.5/99.97%

100%/99.5

60-150ms (reg) 150-350ms (global)

1-20ms

Video

99.5/99.97%

>95%/99.5

50-100ms (reg) 150-350ms (global)

1-20ms

Audio

99.0/99.95%

100%/99.5

80-200ms (reg) 200-500ms (global)

1-20ms

Data

99.0/99.95%

100%/99.5

80-200ms (reg) 200-500ms (global)

1-20ms

Video

99.0/99.95%

100%/99.5

80-200ms (reg) 200-500ms (global)

1-20ms

Delay (One way)

Near Real-Time

Background/Batch

Table 4.3: Multi-media service characteristics.

You will also note that application availability measurements will actually decline as one goes from real-time, at 99.7% actual and 99.99% SLA, to background. The reason is that in a contention situation, where available resources are below normal and there is competition for bandwidth, buffers and processing capacity within the network, real-time applications should receive highest priority access, near real-time next priority, and background/batch lowest priority.

89

Chapter 4

Delivery/Accuracy/Loss The second consideration of application quality measurements will be a category that can be referred to as delivery, accuracy or loss. If you see the glass as half full, delivery will be your choice of terms as it represents the percentage of packets that were sent from the source that actually make it to the destination. If you see the glass as half empty, your chosen term is loss, as you would think in terms of the percentage of transmitted packets that never make it to the destination. If you really don’t think about glasses as being half full or half empty, you might consider this value as the accuracy of the network in delivering packets. Let’s take a positive posture and consider the percentage of packets delivered. As with application availability, you usually see SLA values for telephony and video that are greater than the actual expected minimum values, except in the case of data and all varieties of background and batch services for which expected delivery is 100%. The reason for this is that the later services make use of the Transmission Control Protocol (TCP) as opposed to the telephony and video’s use of the User Datagram Protocol (UDP) for sending information across the network. This distinction is important because of the way in which a transmission error is handled in the network. Basically, TCP will retransmit if errors are encountered and UDP will not. The reason why the 100% actual number is less than the SLA value, therefore, is because the SLA value counts the percentage of packets that are sent that arrive on the first attempt. The 100% value may require additional attempts, up to some retry limit, but the 100% number is used for planning purposes: 100% of sent packets arrive at the destination, and the retransmission time for any packets that need to be sent more than once due to errors while traversing the network is adjusted for in the overall delay time. It is noteworthy and important that IPT uses UDP for voice transmission and in most implementations for signaling as well; therefore, UDP never retransmits voice or signaling packets. If they are lost, they are lost.

90

Chapter 4

Error Handling in Packet Networks Packet networks are not perfect. There is a variety of reasons why packets may be damaged—that is to say, have the value of their 1 and 0 bits changed—or for packets to be discarded and never arrive at their intended destinations. This is as true of data packets as it is of packets carrying voice or video, but the way of handling the problem is a bit different, depending upon the protocol being used to send and receive the information.

Bad Packets To assure accurate transmission of the packets, the sending system does a special calculation on the bits in the packet. The calculation is usually referred to as a Cyclical Redundancy Check (CRC), block check, Frame Check Sequence (FCS) or Checksum. These are all variations on the same theme used in different transmission protocols. The results of the special calculation are either placed at the end of the packet—in the packet’s trailer following the payload bits being transmitted—or in the control bits at the front of the packet called the header. In either case, they are intended as a special part of the message that allows the receiving system to determine whether any 1 bits were changed to 0 bits, or vice versa, which could drastically change the meaning of the information being transmitted. When the packet arrives at its destination—or in many cases at an intermediate relaying system, such as a router—a calculation identical to the original calculation is performed again. The new result is compared with the result of the original calculation stored in the packet by the sender. In the case that the new result and the original are the same, it is assumed that no bits were changed and the packet is either forwarded or used by the system. In the case that the new calculation does not match the original calculation, one of three outcomes is possible. In some cases, the error checking calculation is so sophisticated that it is possible for the receiving system to determine which bit or bits were changed, up to two bits, and fix them. In other cases, the receiving system requests the sending system to resend the packet, up to some number of retransmission attempts as identified by the retry limit. In still other cases, the receiving system discards the inaccurate packet and lets some other part of the network worry about the missing information. Lost or Missing Packets It is also possible that packets may be discarded or get lost along the way for other reasons. If, for instance, the intermediate routers between the source and destination become overburdened they might intentionally make a decision to discard packets. This situation, usually referred to as network congestion, is not uncommon at times even in well-architected and managed networks because building a network to handle every possible demand is cost prohibitive. In the case of lost packets, some protocols will provide a sequence number for transmitted packets so that receiving systems can know whether packets are missing. Some systems, however, do not. In the case that a sequence number is missing, packets can be recognized and a request can be made that the missing packets be retransmitted, up to a specified retry limit.

91

Chapter 4 Which Protocols Do What? The Internet Protocol (IP) is a protocol common to all applications in IP networks. IP is always used in conjunction with a higher-layer protocol, which is virtually always either TCP or UDP. IP has an errorchecking mechanism, but only on its header information, and has no sequence number capability. IP relies on the higher-layer protocols to provide sequencing, if needed. TCP is a richer, more robust protocol and has both an error-checking mechanism on its header and a sequence number mechanism, with the tradeoff of more processing and higher bandwidth utilization than UDP. UDP has neither. For these reasons, TCP is used when information accuracy is vital, such as paychecks and browser info, and UDP is used for IPT and similar applications.

Delay Delay figures are cumulative numbers that include all factors that can impact the time it takes a packet to travel from sender to receiver. In addition to the delay due to the network distance of a transmission over physical links from sender to receiver, there are added delays for the time that packets spend in intermediary relaying systems such as switches and routers, known as nodal delay. Delay is usually a constant factor for any given communication. Reasonable ranges of values, for planning purposes, are shown in Table 4.3. Both regional (same continent) and global (different continent) values are shown. The component of delay due to distance is usually fixed or so very close to fixed for most connections that it can be considered fixed. In addition to planning, these numbers are equally valid for SLA purposes, though some service providers may show more accurate numbers for different continent pairs. Delay Variation or Jitter Delay can also have a variable element due to a variety of factors, such as how long the packets spend processing and queuing in intermediate systems on their way from the source to the destination. As has been noted earlier, for the 100% delivery applications, delay variation might also be impacted by the one or more retransmissions needed to achieve the 100% delivery. Delay variation is usually absorbed for data applications and is usually unnoticeable to the end user, while delay variation in voice applications can be very noticeable and contributes to user complaints and dissatisfaction. Delay variation is shown in milliseconds and is rarely less than 1 or more than 20. These values may be used for network design as well as for SLAs. Fixed delay occurs in traditional telephony systems and can be quite noticeable in transoceanic cable or satellite systems. Delay variation also occurs in traditional telephony but is usually so negligible due to the high-speed circuit switches that provide the foundation for older networks that delay variation due to intermediate switching systems can usually be ignored.

92

Chapter 4 In all the real-time and near real-time applications that Table 4.3 shows, it is assumed that there is a human watching or listening for the information to arrive, and therefore, in terms of delay, near real-time is in a close second place behind real-time. In real-time applications, end-to-end delay must be very short and with as little variation as possible to prevent disruption to the timing and flow of the human conversation. It is interesting to note, almost contrary to what one may think, that very long fixed delays, even exceeding a second and a half, have a less negative impact on MOS values than do much shorter but inconsistent delays. Variable delay is disruptive to the pace of the conversation but once the speakers adjust to a fixed delay, such as that experienced over a satellite link, they are much more comfortable speaking. Near real-time applications, however, can tolerate greater delay, though not the excessive delays possible in background and batch applications. In the case of near real-time applications, buffers may be used to optimize throughput without introducing delay that lowers the QoS perceived by the human user. This is not to say that buffering information for near real-time applications is always an acceptable solution. Differentiation, Prioritization and Queue Management The key to optimizing all applications is to provide the proper mix of resources and handling within the underlying IP network, and the first step in this optimization is proper differentiation. By knowing the specific needs of the specific type of traffic contained in each packet, appropriate and different handling can be applied, increasing the likelihood that all packets will be handled optimally. In general, voice applications are less impacted by loss of information than data applications. Studies have shown that as much as 10% of voice samples can be discarded randomly from a voice conversation with little or no loss of understanding, although a change of a single bit in a data transmission can drastically alter the meaning. Due to differentiation when selective discarding of packets in an overload situation is required, voice packets can be selected and discarded, instead of data packets, up to some limit near 10%. Data applications, in contrast, are less affected by delay and delay variation than voice packets. For this reason, in a situation in which some packets will be delayed and some will be forwarded, differentiation allows voice packets to be prioritized and sent with lower delay, while data packets may be delayed to offset the handling of the voice packets.

93

Chapter 4 There are several mechanisms that can be used in networks to differentiate one type of packet and its preferred treatment, but the mechanism common to most applications is provided by the one protocol common to all applications in IP networks, IP. Figure 4.3 shows the format of the special control bits present in the header portion of the IP packet. The area of most interest to the current discussion is the Type of Service (ToS) Byte, pronounced “toss byte” and more recently described as the Differentiated Services (DiffServ) field. The network may also be configured to use Explicit Congestion Notification (ECN), which uses two bits in the DiffServ field in the IP header. The ToS bits provide four types of information: the precedence or general priority of the packet, and within the precedence, normal or low delay, normal or high throughput, and normal or high reliability. Precedence has eight levels from “routine,” the lowest priority, to “network control,” the highest priority, and six classifications in between. By setting these bits, an application such as FTP or IP telephony, can signal the network as to what special treatment the packet requires. In network design, it is imperative to assure that the ToS bits are set properly during modeling and simulation exercises and to be sure that desired values find their way into the final network implementation. It is also important to note that although ToS bits are defined relatively clearly in “standards,” there are different interpretations of ToS bits from manufacturer to manufacturer and even from one product line to another, even though the IEEE 802.1p and 802.1q standards clearly and unambiguously describe how to map the Layer 3 DiffServ Code Points into Layer 2 Virtual LAN labels, and vice versa. There are, therefore, ramifications for networks in which packets might encounter different interpretations on their path from end to end.

Figure 4.3: IP header structure.

94

Chapter 4

Prioritization in Perspective In the mid 1990s, a major global telecommunications carrier implemented a new multi-media VPN network for a large multi-national customer. The carrier had sold the customer on using their multi-media VPN service as opposed to either building it themselves or using a competitor’s network, based upon the carrier’s superior ATM platform and sophisticated prioritization mechanisms. The customer paid a premium for certain service classes. Although the carrier actually offered several gradations of class of service, the new customer settled on three, which they referred to as First, Business and Coach using an airline analogy that is fairly commonplace in networking. Each class of service had specific characteristics associated with it, and each one had a different cost per megabit of information transferred. The user organization accepted this pricing arrangement reluctantly after being sold on the benefits of class of service and the positive impact it would have on application performance and the resulting end-user productivity. After the network was installed and operating, the customer decided to test the differentiation characteristics to ensure that they were getting what they paid for. They set the appropriate parameters on a packet generator and tried the First Class traffic. Indeed, they did get the promised performance and were very pleased with the outcome of the test. They then set the parameters on the packet generator to simulate Business Class service. The customer was a bit perplexed, and not particularly happy, when they found that the traffic got the same high performance characteristics as the First Class traffic. One might think that they should have been happy to get the same performance for a lower price, but they were unhappy because they felt that the reason for paying the higher price was bogus. They repeated the test for the Coach Class traffic and got the same results. What were they to do? There were some among the group that proposed sending all traffic as Coach Class, which, as their tests had shown, would give them performance equal to that of First Class but at a Coach Class price. There were others who proposed sending all traffic as Business Class, in case there was some trick they had missed. At least they would not completely ruin the performance but could save some money. The problem was that marking all the traffic as any single class, regardless of the treatment of that class, defeated the purpose of differentiation and made prioritization unnecessary. In the end, they called the carrier, explained their test, and asked the carrier to explain the lack of difference between the classes. The carrier explained that their network backbone was actually over-engineered and that the differentiation, although always present, and the associated prioritization, while always applied, would only become evident if there were a competition for resources that would only occur during a network outage in the backbone. The customer felt, in effect, that what they were paying for was an insurance policy in the unlikely event of a service-affecting outage in the carrier’s network. And, they really were correct. The customer was able to renegotiate their contract with a much smaller price difference between First, Business and Coach classes and is now very happy with their multi-media VPN. It is also noteworthy that the customer has never had to make any “claims” against their “insurance policy,” but now that they understand the issues and the cost of the “premiums” is lower, the customer is very pleased with its situation, especially as they migrate more IP-based voice onto their network.

Voice QoS QoS is a measurement of the treatment of the packets traversing a network and includes delay, delay variation, packet loss and network availability. Classification and marking of packets, in an attempt to assure certain promised levels of quality, is done at or near the network edge and are controlled by Class of Service rules. The calculation of instantaneous QoS in the network, while important, provides only one measure of the user’s quality of experience. With its broader definition, QoE is rapidly becoming the more important consideration for voice services.

95

Chapter 4 QoE QoE is defined as a telephony system user’s perception of the quality of the communication being experienced from both the technical QoS rating and other aspects of the call. QoE takes into account the cumulative effect of network characteristics on the human speech being transmitted and received. QoE was first designated as a methodology for assessing the satisfaction of users of network video services in the mid 1990s and has been applied to voice since the late 1990s. QoE is a result of a variety of factors and is best measured by way of a human MOS, which represents the opinion of a panel of human judges, on a five-point scale with five being the highest. Due to the loss in translating analog speech waves into digital values and back, in effect, 5.0 MOS has never been achieved. The best Pulse-code modulation (PCM) scores are in the 4.2 to 4.4 range, depending upon network conditions, but are consistently 4.4 in ideal lab conditions. In this early phase in the network design, it is often desirable to convene a panel of judges from your user community and perform MOS panels to get user buy-in and to assist in the process of properly setting expectations. This approach is an important tool for setting proper user expectations and assuring that the system will meet user needs but becomes impractical after implementation. After implementation, and to a great extent during design and benchmarking, automated tools will be used to simulate certain types of calls, estimate what the human MOS would be and help isolate the factors that are impacting voice and call quality. Even using PCM in the new IP-based system, the voice will sound different, but the idea is that “different is not bad and, in fact, a properly tuned IP voice system can actually be better than old analog and digital voice. Care should be taken in the selection of panelists to choose from the broadest range of job functions in order to optimize the voice coding selection for your specific network. However, because it is not possible to empanel human judges whenever an opinion on voice quality in the actual operating network is desirable, a number of measurements and derivative calculations have been developed (as described later in this chapter) that will form the basis not only for design but also for the ongoing network measurement and reporting and will be specified in the SLA. This function must be automated using tools developed specifically for the monitoring and reporting of voice and call quality issues. Tools such as those used to monitor router or switch performance, packet loss, delay and delay variation, and system availability only provide a very general idea of the effect of network activities on voice and call quality. QoE vs. QoS Although QoS is an important element for technical issues such as SLA compliance, meeting users’ QoE expectations is crucial to a successful IP telephony implementation. QoE can be affected by many factors: •

Human factors

•

Voice encoding/compression

•

Network issues

•

Service/feature support

96

Chapter 4

Human Factors

The biggest factor impacting QoE is user expectations. The users’ prior experience and use of telephony will play a large part. Are they used to a highly reliable traditional desk phone and are being asked to use an IP system that occasionally drops calls or a wireless or cordless phone that does not work at certain locations in the building? Is there a critical path or lifeline application such as 9-1-1 emergency calling or do they use the phone for revenue generation, such as an outbound sales call center? People who use the phone for casual conversation or other less stressinducing purposes are likely to be less harsh critics of any new system. Speaker and listener age and gender also impact QoE. Age has been shown, for instance, to impact MOS. Older listeners tend to give higher MOS while younger listeners’ MOS values tend to be lower, which is almost counter-intuitive. One would think that those who have been listening to phone systems longer would be tougher critics than younger folks, but research shows that older listeners have more highly developed linguistic ability and can get more information from less communication; therefore, they are more satisfied with the same signal than younger people. Gender also impacts MOS. Males tend to give higher MOS and females give generally lower MOS. The primary reason for this seems to be that higher frequency sounds are more important to female communication, and these signals are often lost or distorted in the analog- to-digital conversion process. Speaker and listener familiarity also plays a role in QoE. The more familiarity between the speakers, such as a mom and child, the lower the MOS; the less familiarity, such as a first time caller to a call center, the higher the MOS. Native language of the speaker is also an important consideration. Although PCM-based systems get fairly consistent MOS regardless of the native language of the speaker, due to PCM’s analogto-digital conversion process, systems based on Code Excited Linear Predictive (CELP) are often optimized and get lower MOS values if not optimized to the unique sounds inherent to the native language of speaker. This is because CELP-based systems are voice-optimized and use code books containing sounds that are matched to codes for transmission. In order for a sound to be transmitted, it must have a corresponding code. These considerations are often very difficult to factor into a design, but are critical to user acceptance and satisfaction and must, at least, be considered. Use of PCM for voice coding and reducing the number of recoding at gateways and other points in the network will improve the MOS and the user’s perceived QoE.

97

Chapter 4

Voice Encoding/Compression

One of the underlying assumptions going into this network design effort was that in situations of bandwidth versus voice quality, voice quality will prevail. For this reason, IPT systems should be implemented using the G.711 coding scheme, known to all as PCM. PCM is the standard for voice coding worldwide in traditional telephony networks and although it has not been implemented quite as widely in IPT networks, there are a number of desirable characteristics to recommend it for the job (see Table 4.4). ITU-T Standard

Coding Scheme

Bit Rate

Sample Size (Bits)

Encoding Delay (Time)

Mean Opinion Score (MOS)

G.711

PCM

64K CBR

8 bits

300ms will impact the users’ QoE. Delay should be specified in the SLA.

•

Delay variation (aka jitter) is a bigger QoE problem than delay. Delay is fixed and easier for the listener to adjust to. Delay variation is difficult to adjust for and causes anxiety and stress on the part of listener. Jitter less than 20ms one-way is ideal and should be specified in the SLA.

•

Packet loss >1.5 to 3% is where many human listeners can begin to perceive a degradation in voice quality even though the human ear can adjust to packet loss up to about 10%, or more in certain circumstances, such as where the speakers are known to each other or there is a known call context. Packet loss impact can be reduced by shortened voice samples and fewer samples per RTP packet, which is something that the organization responsible for programming the IP phones or adapters has control over. The end-user or end-user organization may or may not exert control, but this should also be addressed in the SLA.

•

Network availability directly impacts QoE. No dial tone is a BIG problem and causes loss of confidence in the system. This needs to be a key component of any SLA.

100

Chapter 4

Service/Feature Support

In migrating from an old system to a new one, it is critical to ensure that the new system performs all needed functions of the system being replaced and, to the extent possible, that users are comfortable with and accept any changes to work procedures required by the new system. This may seem like an obvious and simple point, but user reluctance to change is one of the biggest reasons for the failure of IPT systems. A number of valuable considerations can be made during the network design phase. For instance, is Voice Band Data (VBD), such as analog modems and fax machines, to be supported in the new network? If so, it will be necessary to test modems at all speeds and verify all fax groups work without down-speeding. As simplistic as it may sound, it is necessary to test Dual Tone Multi-Frequency (DTMF—aka Touch Tone) systems. DTMF is critical for many applications and not all voice coding algorithms support all digits. An inventory of all calling features, as described in Chapter 3, must be performed prior to the network design phase to ensure that allneeded calling features are supported. Supervisory tones and intervention, signals that allow the network to perform special functions, such as allowing a person to break out of a voicemail system or make another credit card call, must be supported as well as all call forwarding, routing and blocking features and any other calling features identified in the inventory. Billing and call accounting must not be ignored, abandoned or otherwise neglected during the network design phase. Many organizations use call accounting and billing for a variety of reasons, such as customer billing, verification, tracking of calls, and allocation of internal costs. This topic is covered in more depth later in this chapter.

Voice Metrics and Measurements There are two basic types of voice quality testing, non-intrusive, often called passive, and intrusive, often called active. Non-intrusive does not require special test calls and is based on actual conversations. Intrusive voice quality testing requires a special test call and is based on a comparison of source signal with signal after transmission. The most desirable situation is to assemble a panel of judges and let them listen to a call and give the call a score from 1 to 5, with 5 being the best. After initial selection of products and technologies that will be put to use in the network, however, this approach is impractical; therefore, a number of calculated values are generally used by automated testing tools, including MOS, J-MOS, PSQM/PSQM+, PESQ/PESQ-LQ and R factor. Non-intrusive methods include MOS, derived MOS, E-model and R value/R factor. Intrusive methods include Perceptual Analysis Measurement System (PAMS), Perceptual Speech Quality Measure (PSQM) and Perceptual Evaluation of Speech Quality (PESQ). The following brief review will allow you to understand their relative importance and value in the network design process.

101

Chapter 4

MOS

MOS is adopted from traditional telephony. The traditional “opinion” of voice quality in the U.S. was from a panel of 16 human judges from Central Illinois consisting of 8 men and 8 women who judged voice quality on a scale from 1 (lowest) to 5 (highest). This type of humanoriginated MOS is useful for human user QoE analysis but is useless for SLA development and network monitoring. Today, MOS is often calculated by a management tool using the QoS indicators for loss, delay and jitter. PAMS

PAMS was developed by British Telecom and does not replace MOS, though it was a good first attempt at an automated MOS estimate. PAMS compares original analog voice waves with reproduced/transmitted speech using a complex weighting method intended to take into account characteristics important to the human ear. The scale is from 0 to 6.5, with 0 being “perfect” (no difference between waves). PAMS values estimate MOS +/- 10 to 20% and therefore do not provide enough accuracy for use in IPT networks though PAMS values are excellent for benchmarks and comparisons when both comparison values are calculated using PAMS. PSQM/PSQM+

PSQM is defined in ITU-T Recommendation P.861. Like PAMS, PSQM and PSQM+ do not replace MOS, though PSQM more closely estimates MOS than PAMS at +/- 10%. Like PAMS, PSQM and PSQM+ are excellent for benchmarks and comparisons and represent improvements on the PAMS algorithm. Like PAMS, the scale is from 0 to 6.5, with 0 being “perfect” (no difference between waves). PESQ/PESQ-LQ

PESQ is described in ITU-T Recommendation P.862. Like its predecessors, it does not replace MOS (+/- 10%), is excellent for benchmarks and comparisons, and is an improvement on the PSQM algorithm with the same scale from 0 to 6.5, with 0 being “perfect” (no difference between waves). E-model/Design Tool

E-model, described in the ITU G.107 standard, is a predictive tool that is excellent for use in modeling voice quality in a packet network. The use of design tools incorporating the E-model algorithm combined with network performance data gathered from the live, base network is strongly recommended. E-model predicts average voice quality using a sophisticated and mature mathematical model that takes into account delay, delay variation (jitter), packet loss and codec performance. The result of the E-model calculation is an R factor/R value that predicts voice quality on a range from 0 to 100, where 0 indicates lowest voice quality and 100 indicates best quality. R factor Emodel scores are most useful when based upon measured, rather than hypothetical, parameters. Analysis should, when possible, include Real-Time Protocol (RTP) streams, source and destination addresses, sequence numbers and jitter profile in order to most accurately predict the impact of the IP bearer on MOS. Figure 4.4 shows the relationship between the MOS value and the R value resulting from the Emodel calculation with a description of the user satisfaction level that they represent. A G.107 default value of 94 corresponds with the MOS value of 4.4, which is a target value for many network design efforts. 102

Chapter 4

Figure 4.4: R Value compared with MOS.

Call Integrity/Privacy/Security

Migrating voice calls from traditional telephony networks that are inherently secure, particularly within the backbone of the network, to shared IP networks that are notorious for their insecurity requires a new level of diligence in call integrity, privacy and security not previously seen in telephony. The idea that IPT is just another application is one that has been taken to heart by hackers and cyber-criminals and the archives are beginning to fill with hacker exploits against VoIP, IPT and VoP. The key points to keep in mind during the design phase are that IP phones are not traditional phones with a keypad and a ringer. They are special-purpose computers in a telephone form. They have processors, memory and protocols and can run programs— increasingly programs written in Java or formatted using eXtensible Markup Language (XML) and similar languages that are familiar enough for the hacker to compromise and utilize. In all phases of the network design phase, physical and human security must be considered and, like any IP networking project, the IPT services and the systems that support them should be subject to periodic security audits and penetration tests. Impact on Business Processes In the best possible case, the migration to an IP-based voice platform will have an immediate and positive impact on operations, costs may be lower, and users will be ecstatic with the new applications and Unified Communications tools provided by their wonderful employer. In fact, the new system will give them bragging rights beyond their wildest dreams. IP phones on their desk and calls over WiFi phones that attach to their hip or fit in their purse. It can’t get much better than this! But, alas, the truth is probably a bit more mundane: this, after all, is dial tone we are talking about. Your reality will, of course, probably be somewhere between these two extremes, and it will be just as important to capture positive impacts and socialize them within the organization as it will to identify negatives and quash them.

103

Chapter 4 Standardizing Positive Impacts Standardizing positive impacts can be accomplished through two channels: formal training and a grass-roots effort. Formal training can be via classroom, webinar or written and can be conducted all at once or phased over time with practice in between. The grass-roots approach can be as simple as featuring employees who have put the new system to good, company-approved uses to demonstrate their skills at informal “lunch and learn” sessions or in the context of other meetings. Many organizations really do take the “it’s just dial tone, what could be so complicated?” approach and neglect training, but they do so at their own peril. Absent official training, users will develop their own skills and habits and may or may not make a positive contribution to getting the most productivity out of the new investment. “Over-the-cube wall” help should not be a fall-back to official training. Avoiding Negative Impacts Negative impacts can be avoided by having a simple, clear system for reporting problems and a feedback mechanism to let the users know that the problem is being investigated or has been resolved. Most organizations choose to use their existing Help Desk and trouble reporting and tracking system. Any organization considering this approach should be sure that their Help Desk personnel are properly trained and that their systems are capable of handing the volume and issues.

Translating Needs to SLAs At this point, it does not matter whether the organization’s new IPT services will be implemented in-house, outsourced or some combination. In any event, an SLA will be needed. The SLA will become, effectively, the voice user’s Bill of Rights and will represent what the user has been promised, how compliance is measured, and any penalties that might exist for non-compliance. In a growing number of cases, users are also held accountable for their actions in terms of number and types of calls, destination, use of gateways into traditional telephony systems and other actions, and are monitored under the agreement.

104

Chapter 4

The “Proper” SLA There are many different types of SLAs in the wide world of networking and telephony, and many of them are “proper.” In order to be a “proper” SLA, the SLA must not only spell out the requirements of the carrier or service provider (or internal IT organization) and the attendant penalties but also specify similar requirements and penalties for the customer organization (or business unit). It is also common to specify certain benefits or bonuses for exceeding the requirements set forth in the SLA. These provisions acknowledge the fact that in certain cases resources are limited and encourage the carrier, service provider or IT organization to apply those limited resources to the problems of a specific customer or business unit. An SLA is often an addendum to a service contract or a contract in its own right, and this should be considered from the outset. Because the document may at some time in its future be evaluated and interpreted by a judge or arbitrator, attorneys’ technical terms, acronyms and jargon must be clearly defined or eliminated entirely where doing so does not hurt the meaning of the document. Nothing should be left to interpretation, diagrams should be provided, and, where formulas are used, examples should be provided as to their proper application. Terminology should be clear, crisp and unambiguous, and nothing should be left to chance. If it is understood, for instance, calculations of service availability assume a maintenance window from midnight Saturday to 2:00 am Sunday, that understanding should be explicitly stated and an example of the service availability calculation should be provided. Metrics The proper SLA may include any number of items, but at a minimum should include specific details for application availability, packet delivery, delay and delay variation. For numerous guidelines on specific measurement details, refer back to Table 4.3. Chapter 3 also provided some guidelines for other possible aspects for a proper SLA, including throughput, call setup and teardown boundaries, and MOS values. Availability should include any distinctions for on-net, off-net, wireless access or other variations as well as allowances for time zone differences, maintenance windows and other periods of unavailability that are known in advance and should not be included in down-time calculations. Availability that differs per application should also be noted, as well whether availability is for a single user, class of user, department, geographic region or network-wide. Where possible, the availability should be designated for a site too. In very large networks, it is possible to have a single user or site down for a very long time while hundreds of sites or individuals are up and running. Packet delivery should include any special considerations for retransmission protocols, variations in requirements by application or class of user, geographic area, wireless vs. wireline and so on. Special consideration may also be given to packet delivery during certain network outages or other special circumstances. Delay and delay variation targets in SLAs can be very general but should really be at least at a continental and inter-continental level for all appropriate continent pairs to which service will be provided. This type of tiered SLA for delay and delay variation provides more accuracy, better planning and a clearer understanding of voice impairments when they are reported as well as a better understanding for remedies under the SLA in case of non-compliance.

105

Chapter 4 Other items that merit consideration for inclusion in the proper SLA are response times and Mean Time To Repair (MTTR) commitments—not just “mean time to respond”—for systems and users under contract and those not under contract and for those within certain distances of service centers or service personnel. Consideration should also be given for customer-provided on-site stocking of spare components or use of hot-standby components to lower costs, increase uptime, and alleviate some of the pressure on service staff for the health of the network. Time windows for upgrading existing service and for adding new sites to the network should also be spelled out clearly in the SLA. In all cases, measurements and penalties should be clearly stated and examples shown so that a “Did you comply with section…”-type questions can be answered with an unambiguous “yes” or “no” and the veracity of the answer can be known. Measurement Measurement of all compliance metrics, to the extent possible, should be automated and have thresholds for automatic exception reporting that match those found in the actual SLA. Exception reporting is highly desirable as it cuts down on the total number of alarms and alerts and allows you to focus only on those issues that represent non-compliance with the SLA. Care should be taken to align reporting with local time zones and units of measure, as well, to allow the fastest understanding of SLA compliance reports and most intuitive and localized approach to problem resolution. Where possible, SLA penalties should also be automatically calculated and credits applied, as appropriate. Measurement, specifically for SLA compliance, is a very important area to apply IPT management tools. The reasons for this are numerous. From a business standpoint, costs are reduced and consistently can be assured by using an automated tool that is aligned to the specific, agreed-upon metrics of the SLA. Any manual approach or one based on lower-layer network statistics from which guesstimates of the impact on voice and call quality are calculated lack efficiency and consistency. From a technical standpoint, the sheer volume, range and interaction of measurements needed to develop meaningful reporting and to highlight areas for problem isolation and troubleshooting are daunting and could easily overwhelm any semiautomated or manual system. Combining voice-aware or multimedia-aware IPT management solutions with an exception reporting approach and trend analysis to anticipate problems and eliminate them before they occur, as opposed to simply trying to mitigate them when they do occur, should be the objective of every IPT network design and implementation. It is far easier to design the proper IPT management into the system at the beginning than it is to go back and retrofit a solution later. Establishing Feedback Loops and Review Cycles Feedback loops allow operational data to be gathered from the live network and compared with SLA requirements. Adjustments then can be made to the network. Early in the roll-out of a network, after final system test and acceptance, review cycles should be performed frequently, possibly daily, and a conference call or meeting involving operational personnel and management should be held. After the network settles down, weekly then monthly meetings should be conducted. The review meetings are a part of the ongoing network operations and optimization cycle. Review meetings should be held at least monthly to ensure that the network is operating as desired.

106

Chapter 4

Validation of Design Many organizations prefer to choose a network design and validation tool in the early stages of a project and to use the input forms and screens of the tool to provide guidance as to the information that needs to be collected. Others prefer to determine the important characteristics of their network and then choose a tool that fits their needs. This guide takes the approach that the choice of the tool should be dictated by the need; therefore, it is at this point in the process to select the design and simulation tools that will be used in this design phase. A report card similar to the one provided in Chapter 3 for assessment tools is provided with this chapter to assist in the choice of network design and validation tools. Very often, the same provider will have tools for assessment and for design and validation that use the same inputs and produce results with little additional effort. The version of the Modeling, Measurement, Monitoring and Management Tools Report Card (4MTRC) for network design and validation will assist you in the selection process of these very important tools. The report card is a set of five Microsoft Excel spreadsheets. There are five identical report card spreadsheets that may be used to evaluate up to five separate 4M tools. The sixth spreadsheet allows you to compare the tools side by side and support an intelligent selection process. The spreadsheets are locked to avoid inadvertent changes to cells containing formulas and labels, but a password is provided so that the spreadsheets may be modified by an organization for its own use. Those knowledgeable in the inner workings of Excel can take a look and see that there is nothing particularly clever about the coding. The real value is the knowledge that is embedded in the categories that were chosen for comparison as well as the weighting factors applied to the different categories. For users with limited resources, these two factors—the comparison list and associated weighting— will be an invaluable source of assistance in the selection process; for the more resource-rich organization, the spreadsheets will provide a convenient starting point for building customized selection report cards for your organization. The first project will be to gather together all the Excel spreadsheets, Microsoft Word documents, Microsoft PowerPoint presentations, handwritten notes, and the appropriate sections of the SLA that were generated during the early project specification stages and put them into the network design tool. At this point, the design team can create an operational baseline, validate the baseline against real-world observations, then commence the process of network tuning. The cycle will include modifications to the design and service order, the SLA and, possibly, management and user expectations. It should also be possible to begin tracking costs in real-time as network elements are acquired. Network Tuning After establishing the baselines and validating the model or simulation results against observations of the real network results, it is possible to perform network tuning, which is an optimization process in which specific characteristics of the network are systematically modified to minimize or maximize certain characteristics. The most important measurement in the network tuning phase will be the E-model R value. Every effort should be made to maximize the R value while keeping cost in balance, both in terms of network costs and resource utilization. Before beginning any serious modeling effort, engineers should familiarize themselves with the tools and just “play around” to gain familiarity, posing hypothetical questions and answering them, and modifying parameters at will. After the actual network tuning exercise begins, each step should be documented and only one change should be made to the model at a time.

107

Chapter 4 Test Bed Architecture The full network, including all other IP traffic and systems, should be modeled during the simulation phase. If it is impractical to do so due to the gargantuan size of your network or other restrictions, at least model all the elements of a sub-network of your network. After the modeling or simulation on the computer, it will be necessary to build a small test bed in a controlled laboratory and, after appropriate testing, to move the test to a real environment. Modeling vs. Simulation Modeling and simulation are closely aligned and the terms are very often used interchangeably, but while the objectives are very similar and the results may look the same on the surface, the processes underlying modeling and simulation are very different. Modeling and simulation are both attempts to predict performance of network-based services given some set of inputs. Inputs can range from wild, “seat of the pants” guesses to sophisticated traffic studies. The fundamental difference between modeling and simulation is that modeling provides a snapshot at a moment in time while simulation is a process over time with changes to network traffic patterns, call volumes, Class of Service (CoS) requirements, and QoS and QoE results. The budget-minded very often use an Excel spreadsheet to develop reasonably accurate snapshots, are emboldened by their results, and use their informal models for actual network design, validation and optimization. This approach appears savvy and does save costs in the short term, but even small variances, which are not even really mistakes or miscalculations, can magnify in the real network and can be very costly, often costing many times the price of a good simulation tool, training and some consulting to get the project kick-started.

There are several issues that need to be considered and tested based upon your environment. Considerations and parameters that are common to most IPT projects include: •

Network impairments

•

Timing/synchronization issues

•

Buffer management

•

Static/dynamic/adaptive jitter buffers

•

Broadband bandwidth measurement and calculation

•

Gateway performance

•

IP device performance

•

Feature support and end-to-end operation

•

Scalability

Network Impairments Network impairments should be introduced in the test bed architecture to simulate characteristics that will be encountered in a live network environment. This will be accomplished via software during the design validation phase and via hardware simulators in the lab and limited live beta testing. Impairments that should be considered are testing of extremely long distances, excessive packet loss, excessive delay and delay variation, both individually and combined. All codecs that may be used in the network should be tested as well as all IPT devices, including gateways and phones and adapters.

108

Chapter 4 Timing/Synchronization Issues Although IP networks are inherently asynchronous systems, sending packets of any length when the packets are ready to be sent, success with voice applications lies in ensuring that packet playout at the end systems closely mimics the synchronous delivery of serial voice streams from older telephony systems. This steady voice communication can be simulated very effectively, even in a highly asynchronous IP environment, using buffers and, even better, using the Real Time Control Protocol (RTCP) along with the Real Time Protocol (RTP). Doing so provides a closed feedback loop between the receiver and sender based on information about the arrival statistics of multi-media information being sent. RTCP is not widely used today by enterprises, or even many service providers and carriers. Many organizations are simply struggling to get IPT up and running in any condition, but RTCP will be used increasingly as the focus turns to QoS and QoE. The good news is that most manufacturers are ensuring that RTCP is implemented in their products and that RTCP will be available when organizations are ready to exploit it. RTP and RTCP Multi-media information, specifically voice and video over IP, represent digital information or digitized analog information that must get across the IP network in a manner and with performance characteristics that are much more sensitive to delay and delay variation than most of their data counterparts. Multimedia sessions are established using H.323 or, more predominantly in today’s carrier and serviceprovider networks, Session Initiation Protocol (SIP), and use RTP to actually carry the information. The problem is that RTP alone is blindly sending information into the network with no awareness whatsoever of its fate. Did the packet arrive? If so, what about timing and delivery? What was the delay? How about delay variation? If packets are discarded, is it possible to dynamically modify the transmission, including packet size and content fill, to optimize the connection? If a less-desirable codec is being used for bandwidth savings, is it possible to use a higher-quality codec in the presence of greater bandwidth resources? These issues and more are of high value in optimizing the multi-media experience. Though it is not yet utilized in most networks, there is a solution: RTCP. RTCP is used in conjunction with RTP to provide feedback on the status of the packets as they arrive at the destination, and there are versions of RTCP being developed to provide information about the state of the intermediate network and its performance, as well. Network implementers of IPT should tie RTP statistics back to the SLA.

Buffers can be thought of as “holding tanks” for information as it traverses the network in the form of packets. Buffers are good in that they provide a staging area in the case of momentary network overload to keep packets and reduce loss. Buffers are also bad in that improperly managed buffers can add unnecessary delay to connections, especially time-sensitive multimedia traffic. During the network design validation and testing phases, it is important to manage buffers in end systems such as IP phones and gateways as well as in intermediate systems such as routers. Most routers are not optimized by default for multi-media traffic and often require software upgrades to accommodate true multi-media traffic. The likelihood of a successful implementation of IPT is greatly increased if this optimization is done in an earlier, rather than later, phase. Buffer Management

Most simulation tools will allow some input of information relating to buffer management, such as buffer management algorithms, ingress and egress buffer sizes, and internal switching approaches. Buffers are large areas of memory for storing multiple packets as they transit the network. In some cases, it is valuable to get buffer management performance results from manufacturer-specific simulation tools for input into third-party simulation tools.

109

Chapter 4 Static/Dynamic/Adaptive Jitter Buffers

In addition to the large memory buffers, there are also jitter buffers, which are often as small as three bits in size, that are used to compensate for the timing variations between the circuit and the receiving device. There are three basic “flavors” of jitter buffers: static, dynamic, and adaptive. Static are the least effective for multi-media. They use a single, fixed and predetermined size of buffer, usually optimized for fragmented data packets for all network traffic. Static buffers are very common in older routers, especially the lower-end small office/home office devices. Dynamic buffers are better than static buffers because they dynamically calculate an optimum buffer size based on the first n packets received. Although this approach is preferred over static jitter buffers, it is not an optimum solution. Adaptive jitter buffers represent the state of the art as they can adapt to changing network conditions. Not only should IPT equipment that employs adaptive jitter buffers be chosen for use in the network, the use of adaptive jitter buffers should be simulated to the extent allowed by the chosen testing and validation software tools. Broadband Bandwidth Measurement and Calculation Table 4.5 only hints at the wide range of bandwidth utilization between two codecs of very similar MOS performance, but Tables 4.6 and 4.7 strike closer to the heart of the matter. The difference between Tables 4.6 and 4.7 is simply the packetization interval. Table 4.6 shows packets sent every 10ms, resulting in more packets but a smoother flow of voice that is less susceptible to packet loss. Table 4.7 shows a packetization interval of 30ms. With a 30ms packetization interval, bandwidth is consumed less quickly because more voice samples are placed in each packet, thereby reducing the number of overhead bits per sample. However, the connection is impacted more heavily by packet loss, as the loss of a single packet will result in the loss of three times as many voice samples.

Table 4.6: IPT bandwidth estimate with 10ms packetization interval (Source: Matthew Michels, Nortel Networks).

110

Chapter 4

Table 4.7: IPT bandwidth estimate with 30ms packetization interval (Source: Matthew Michels, Nortel Networks).

Results of this kind can be used to play “what if games” and to develop realistic bandwidth estimates for different types of connections. Gateway, Softswitch, Session Border Controller and IP Device Capacity and Performance Gateway, softswitch/call server, session border controller, and IPT device performance and capacity must also be factored into the mix, especially if the IP devices are not dedicated IP phones. In addition to performance metrics, capacity of the devices must be evaluated and simulated. Metrics of importance for this IPT equipment include not only the number of steadystate connections that are possible to support simultaneously but also the number of Busy Hour Call Attempts (BHCAs) and resulting Busy Hour Call Connects (BHCCs). IPT systems are just now beginning to approach the performance of traditional switching systems and very often it is demand—especially at centralized locations such as PSTN-to-IP gateways and traditional and IP-enabled call centers—that overload systems resulting in lost calls and potentially lost revenue.

111

Chapter 4

Testing, Feedback, Network Modification, Testing Loop Everybody agrees that testing in a laboratory environment, followed by testing in a controlled semi-live “beta” environment, followed by controlled release and then general release, is a good idea. It is remarkable how few organizations actually budget the time and personnel needed to do a thorough job of testing prior to actual live operation. Many organizations simply underestimate the complexity of voice itself—whether packetized or not—and don’t budget enough time or resources to fully understand the system until it is under live load and much more difficult to learn and modify. Organizations will hopefully be willing to learn from the failure of others and budget appropriate resources for the testing cycle as shown in Figure 4.5.

Figure 4.5: Testing, feedback, network modification, testing loop.

112

Chapter 4

Testing/Proof of Concept Objectives There are two main objectives for testing and they will work if the foregoing design effort is done properly: •

Does the theoretical design meet the actual objective?

•

Do chosen products and vendors meet criteria?

And, in case there are either major or minor flaws in the underlying design, this process is set up in a loop to allow modifications to be accomplished at this step that will ensure project success. Feature Support and End-to-End Operation Although many of the capabilities and systems referred to earlier in this chapter can reasonably be tested in isolated environments, all features must be tested end to end. This will begin in the “closed lab” environment, then move into the limited production environment, and then on to the live environment as testing moves closer to a realistic environment. The actual feature set that will be used in your environment will vary, so it is very desirable to establish profiles that include the full set of features a given user profile requires. It is also desirable to have an actual member of the user community who represents other users with the same profile validate the feature set and help test the features. This relationship should have been established well before this point and at this point it should be simply a matter of putting the user to work in helping with final testing. In fact, it is in final testing where the actual users should be put to work. The voice or multi-tech engineers should ensure that features are working correctly and to specifications before actual users are brought in—actual users should be used for validation and certification and not for initial testing and repair. Do not forget that they are the liaison with the other members of the user community—the ultimate judges of the system—and that their opinions on the readiness or non-readiness of the system will filter back quickly. It might be possible to establish as few as four user profiles that are common to all departments, though it is more likely that there will be far more profiles for any given situation. Let’s look at six basic profiles, just to get an idea of the type of feature sets they might need. Although nowhere near the comprehensive list needed by most organizations, this will provide an example with some points of comparison and differentiation.

113

Chapter 4

Profile

Possible Job Functions

Telephony Features

Office worker, general staff, supervisor, low-level manager

Dial tone In-house (extension) dialing Local dialing Long distance dialing Basic voicemail Call waiting Caller ID

Secretary, telemarketer, inside sales person, Help Desk worker

Basic telephony user PLUS: 3-way conference calls Skills-based call routing Call forwarding

Senior secretary, group admin, senior telemarketer, senior Help Desk worker

Intermediate user PLUS: Enhanced voicemail (add fax storage/retrieval and advanced messaging and retrieval features) Enhanced call forwarding Multi-party conference calls

Administrative assistant, senior sales person, senior manager

Advanced user PLUS: Multimedia conferences (Web plus audio) Enhanced call forwarding with remote access

Senior manager, technician, developer, product marketing manager, senior sales manager

Most of the above PLUS: Call routing management and follow-me features Voice messaging Instant messaging Multi-media conferencing Collaboration

Sales executive, account manager, senior sales manager, sales VP, sales director, marketing executive

Remote telephony Remote data Remote video Advanced call routing and followme features

Basic Telephony User

Intermediate Telephony User

Advanced Telephony User

Power Telephony User

Multi-media User

Road Warrior

Table 4.8: Sample user profiles and telephony features.

It is clear that many organizations will have additional requirements, such as various skill levels of call takers, call taker supervisors, operators, executives, administrative assistants, and similar skill groups but this basic mapping of user profiles to features will show a way of simplifying both the testing and implementation.

114

Chapter 4

Closed Lab Environment Initial testing is performed in a closed lab environment following the process described in Figure 4.5. The process begins with testing to ensure compliance with the SLA. At this point, the SLA should be considered unalterable and should be changed only under the most extreme circumstances as it has become the blueprint that has been agreed to by all parties. Test results are compared with the SLA. If they match within defined parameters, it is time for the next step. If not, the network is modified and the testing loop is repeated. After Hours/Off Hours Familiarization and Benchmarking The next step is to perform the same set of steps as in the lab environment in the real network during non-business/non-busy hours. A good practice is to perform this phase of the testing using a staging area. The staging area is clearly marked out with empty tables surrounded by yellow tape on the floor. Everything that is taken into the staging area is inventoried so you will know what tools, software, connectors, test equipment, network equipment, etc. was needed to make the after-hours test a success. From the list of tools developed in this step, it is possible to put together an optimized toolkit for actual deployment and to write up a series of implementation guidelines and field service notes. Busy Hour Testing in a Live Network After success in the non-busy hour testing, it is possible to reproduce the testing during the busy hours in the live network. Many of the principles addressed in Chapter 3 about Network Assessment apply to the live-network testing phase; it may be useful to revisit the “Tips and Tricks for A Successful Network Assessment” section in Chapter 3 at this stage. After successful testing in this controlled environment, it is possible to go to the next step. Evaluation of Results and Ensuring End-User Acceptance Results from all prior steps are now evaluated and a “go/no-go” decision is made. If staging and testing results are not conclusive or compelling changes must be made to the network, testing and validation should be repeated. Otherwise, it is possible to move on to the next step— implementation and migration, which will be covered in the next chapter. One requirement that has been referred to frequently during the first part of this guide as a key to success is end-user acceptance. This point has been discussed peripherally, but it is time to dig in deeper and understand what end-user acceptance really means before you can move forward with the implementation of the telephony project. Prior chapters discussed the importance of keeping the users involved in the project as it progresses and in final testing and acceptance. Chapter 3 went into a great deal of detail about how to inventory the functions of the traditional telephony system to ensure that they are addressed in the new IPT system. And, by “addressed,” not every function of the traditional telephony system must be replicated, but rather, any discrepancy must be dealt with and the end user’s satisfaction must be weighed against other factors.

115

Chapter 4 Let’s assume the laundry list of features has been implemented and checked by the test technicians. At this point, it is time for representative users to be brought in for a final acceptance test: what will they hear, what will they experience and how will they judge the system? The very first new thing with which the user/tester will have to deal is the new phone device. If the new phone is substantially different than the old phone—if, for instance, the new phone has a screen and softkeys or is a PC-based softphone—quick training and familiarization will be needed. The familiarization is best done by identifying the way in which traditional functions are performed on the new phone and then showing any new features that are important to the specific user’s job. What this means is that if someone uses a phone to only make calls, they do not need to be shown how to program the phone with XML. The next step will be for the user/tester to activate the phone. At this point, the user will judge the phone not only on the time it takes for the dial tone to be heard but also on the actual dial tone. Although most people who have never traveled outside their own country don’t realize it, there are actually different dial tones in different parts of the world. The first step to a user’s maximum comfort level and ultimate acceptance of a new system is for the new system to operate in key ways like the old system; the first point of comparison will be the dial tone. In implementations of a new system in a single country, it is important that all new phones present a dial tone that sounds correct for that country. In multi-national implementations, it is highly desirable for all phones to present a dial tone that is consistent with what is expected in each country. The next issue is the time it takes dial tone to be presented to the caller. This is, quite conveniently, called delay-to-dial tone or time-to-dial tone and is one of the most critical elements of a user’s comfort in using the system. For instance, if the user lifts the receiver and waits too long for the dial tone, they will often, almost instinctively, put the handset back on the hook and raise it again to make the call. This time, they get dial tone almost immediately, but it is not a new dial tone. It is the delayed dial tone from their first call attempt. This will happen because dial tone is not automatic, and it is generally not a function of the local phone—dial tone is an audible signal to the caller that they may begin dialing and comes as a result of checking with the switch to determine whether a call would possibly go through if it were dialed. An alternative would be to play an “all circuits are busy now” or other similar announcement. Consider another case. Consider the case in which a caller lifts the receiver and, without waiting for dial tone, begins to press digits. If they pressed 1-555-984-5800, they might get a message that says “a one is required before this number” or even “55-984-5800 is an invalid number, please try again.” What happened? In this case, it is possible that the caller began pressing digits before the PBX or switch was ready, and the PBX or switch only got some of the digits. Both of these problems are related to delay-to-dial tone.

116

Chapter 4 In terms of system design, the time-to-dial tone issue can be exacerbated if an IPT device must wait on signaling from a remote call server or softswitch before issuing a dial tone, which they often do. If the call server is located locally and connected by a high-speed LAN with low utilization, the issue is not much different from a traditional telco arrangement with a local PBX or switch. However, if the new IPT solution uses a centralized call server that is located in a distant office, especially across a congested network and one that is prone to discarding packets, then the Time-to-Dial Tone might be too long, resulting in user dissatisfaction with the new system. There is a subtle difference on how and where dial tone is actually produced in some IPT systems. For example, using a SIP-based system the phone itself may actually generate a dial tone whenever the receiver is lifted or the speaker button is pushed. This initial dial tone is considered a ‘comfort’ feature in that the user expects to hear dial tone. Only after the user has entered some recognized pattern of digits, or pressed a ‘send’ button, does the call server actually get involved in the process of call setup. Equivalent to ‘delay to dial tone’, in this instance it would be ‘delay to call (or session) setup’. The impact on the user is the same; delay is unwanted and is a major factor in call quality and QoE. After the dial tone is received and dialing is underway, the next issue will be any messages and tones that a caller will receive while placing the calls. All messages must be in the proper language and tones must be consistent with what the user expects. This includes any trunk/all circuits are busy and other progress/proceeding tones or network announcements. The most important of these will be the busy or ringing/ringback tone that is received. An issue with the busy or ringing tone involves whether they come from the distant end (remote ringing/busy) or is generated locally (local ringing/busy). In the first case, it is assured that the caller will always hear the appropriate busy or ringing tone for the location being called. This is not always desirable, however, for two reasons. The first one is that even a busy line or a line that is not answered will use network resources to packetize the ringing or busy tone and send it to the caller. The second issue is that the caller is not always familiar with the common tones for all locations they are calling. In this case, the caller may misinterpret the tones. Both problems are addressed by generating the busy and ringing tones locally, at the caller’s end. In this case, bandwidth is used only for the signaling packets that tell the local end that the phone is ringing or busy, and not for the entire duration of the ringing or busy signal. The second problem is addressed because the locally generated ringing or busy tone is always the same—it is always what the caller expects and does not matter to where the call is being placed. Beyond actually setting up the call is the conversation itself followed by call termination. The quality of the call will be judged by comparison with what users are used to. It is often a good idea to establish a set of criteria in advance and to emphasize that “different is not bad” and that the objective is to provide a working, quality telephony implementation but not, necessarily, to replicate the old system in every way. Generally speaking, having the user rate voice quality on the Mean Opinion Score (MOS) scale, with 0 being the worst and 5 the best, similar to the scale shown in Figure 4.4, is the best idea. An E-model R value of 94, which corresponds to an MOS of 4.4, will result in about the same level of user satisfaction as the traditional G.107 Pulse-code modulation. Using MOS and R value also allows correlation of the actual user’s opinion to the metrics used to model the network at which time judgment can be passed on the suitability of the modeling tools being realized.

117

Chapter 4 Fall-Back and Contingency Planning One point that is often overlooked is that success in one phase or step does not absolutely ensure success in the next phase or step; it only makes success more likely. Fall-back and contingency plans should always be a part of the Network Design and Testing phases to ensure that older systems can still be brought back online in the case of a failure. This does not always mean true parallel operation but does mean that nothing permanently fatal should be done to the older system before the operation of the new system has been verified against formal acceptance criteria that have been signed off on by all parties. Use of Third-Party Tools Multimedia networks supporting voice applications are complex organisms requiring specialized monitoring intelligence. It is also obvious that manufacturer-provided monitoring tools are a part of the solution but do not take into account the nuances and subtleties of the overall, usually multi-vendor, environment. However, generic monitoring and management tools are insufficient—to be truly effective, third-party tools must be constructed with some knowledge of the individual manufacturer’s systems being maintained in order to provide a system-wide, endto-end view of system availability, quality and performance. What is needed is an approach that makes best use of key aspects of each. The key areas that monitoring and management tools must cover are: •

Pre-deployment simulation and modeling

•

Manufacturer-specific monitoring and management

•

Real-time business views

•

•

Call detail records

•

Calls in progress

•

Delay-to-dial-tone rates

•

Busy Hour Call Attempts (BHCAs)

•

Busy Hour Call Completions (BHCCs)

•

Gateway channel configuration, utilization and loading

Real-time call monitoring •

Phone and multi-media device availability and monitoring

•

Successful vs. failed call completion rates

•

Delay and delay variation

•

Packet loss

•

MOS

•

Poorly performing components

•

Service level breaches and SLA compliance

118

Chapter 4 • •

•

Summary and exception reporting •

Utilization trends over time

•

Managed devices by company, department and location

Asset monitoring and tracking •

•

Real-time interface to Manager of Managers (such as HP OpenView, EMC Smarts, IBM Tivoli NetCool)

Phone registrations and de-registrations

Capacity planning •

Incoming and outgoing calls

•

Loading by dial plan, routing rules and gateway

•

Bandwidth utilization

•

Delay and delay variation

•

Packet loss

•

Route patterns, utilization and availability

Use of appropriate, qualified, sophisticated third-party tools will allow documentation of results with consistency and reproducibility and a lack of vendor bias.

Summary This chapter provided plenty of food for thought as you enter the network design and predeployment testing phase of your IPT project. It included considerations both from the IP-packet realm and from the realm of traditional telephony. In addition, another report card has been provided to assist you in the selection of design and validation software tools. This chapter has enabled you to put to good use a lot of the very critical foundational knowledge introduced in prior chapters. Chapter 5 will go to the next phase of the project: implementation of the system you have benchmarked, validated and optimized, and then, having established a firm foundation, will begin the task of migrating actual users to the new network.

119

Chapter 5

Chapter 5: Implementation and Migration You have reached the phase in the IP Telephony project in which you will learn for the first time whether your combination of expectation setting, design, and project management is going to be a success. And, because you are working toward a successful deployment, this chapter will point out ways that your project could potentially go astray so that you can avoid those possible pitfalls. To ensure success during the implementation and migration, this chapter will revisit the global enterprise case study mentioned in Chapter 3 and take a closer look at how those involved actually implemented their global IP Telephony project, giving you a blueprint for your own implementation.

External Issues External issues are those factors over which your project team is unlikely to exert much influence, as opposed to internal issues, which will be explored later. It remains to be seen in any given situation which issues will have a more profound positive or negative impact on the overall project, but both will play a role in the project’s success. Reducing Negative Impact of Implementation/Migration There is a recurring theme that must remain in the foreground of your thinking throughout the project—your project, like any other successful endeavor, is being performed for reasons that are acceptable to management with the aim of getting results that somehow improve the business of the organization. This must be true of any project: there are business benefits and goals to be achieved and when you can compare the list of accomplishments to the list of goals and the accomplishments meet or exceed your goals, then, and only then, have you completed a successful project. End users are a group that will often interfere with, or completely disrupt, a project’s success. They can range from administrative staff to executives to network power users whose opinions matter to management. Previous chapters have discussed the importance of getting user participation and “buy in” early in the project—this consideration must continue through every step of the process.

120

Chapter 5 Two of the primary areas for user uprising and resistance during IP Telephony projects are voice “quality” and the new telephone instrument. Both of these issues can be addressed early in the project by setting proper expectation levels and then, in the migration and implementation phases, by getting the user’s acceptance, or sign-off, on the implementation, usually after training. By telling the user what to expect and handling or circumventing disagreements early in the project, project management—using criteria they establish, often in conjunction with user delegates—can truthfully say, “We delivered what we promised.” In the area of voice quality, the issue is rarely “quality” in any way that an end user can actually measure, but, rather, the sound of the voice is usually just different from the old phone system or other phone systems they might use, such as their cellular system. If proper expectations are set early in the project, using tools such as voice quality demos, this should become a minor issue and, in fact, might raise the confidence level of users about things they can’t see, or hear. Much of the value of setting expectations comes from the confidence derived when you actually meet the expectations. The difference between the old, reliable, beloved phone instrument and the new-fangled contraption that is now sitting on the desk after installation and migration is a potential showstopper for IP Telephony implementations. It is possible to ease the transition through training, allowing the end users to become enamored with their new phone device, to allow their opinions to be heard and their needs to be addressed, before they are forced to use the new device. Do not forget that the phone is the basic business instrument of most workers, and problems with the phone, perceived or real, can have a huge detrimental impact on organizational productivity and effectiveness. Absence of a hard “HOLD” button, which is replaced by a soft “HOLD” button that is displayed only when a call may be put on hold, for instance, creates a barrier to use. Disappearance of the buttons that light up on a traditional multiline phone, a different sounding ringer, and similar issues will all contribute to user dissatisfaction and the potential derailing of an otherwise well-designed next-generation phone system. Business Cycles Businesses have natural cycles, which should be well understood and respected. This is an area in which it is useful to consult employees, especially from the IT or traditional telephony staff, regarding past failures and successes. Some things are more obvious than others. For instance, most individuals would be aware enough not to replace a phone system for a United States’based tax accounting firm during the first two weeks of April because the U.S. Internal Revenue Service (IRS) requires individual income tax returns to be filed by April 15th. But, would a similarly aware individual be smart enough to take similar precautions for a tax accounting firm in Mumbai? They would if they understood the business of the Indian firm and realized that this company handled a lot of the tax preparation work from the U.S., which is a point that might not be so obvious.

121

Chapter 5

Holidays Holidays often seem ideal for certain upgrades and cutovers because the workers who use the data and telephone systems are absent. However, it may be equally true that support staff and Help desks are understaffed during the same timeframe, and working on holidays can make for unhappy project staff, which may translate into a higher number of mistakes than might be seen during other times. In addition, in international or multinational support situations, there may be a holiday in the country housing the Network Operations Center (NOC) while it is business as usual in the country where the installation or upgrade is being performed. Coordination A widely known Vice President for a major insurance company once proclaimed, from hard won personal knowledge, that any large IT project required the same skill set as a cat herder. This statement is very true of IT broadly, but especially true for next-generation telephony projects specifically. Like cats, each individual and department that may be involved in the IP Telephony project has their own agenda, likes and dislikes, priorities, and motivations. It is the true professional project manager who can align all individuals and get the desired outcome. Some of the “tips and tricks” are simple in theory but very difficult in practice. Among the tips and tricks are understanding that the IP Telephony project is only one of many activities in which participants will be engaged. Gaining positive participation involves selling the individual on the benefits they will derive by participating, identifying their role as clearly as possible, and limiting their involvement to the very minimum investment of time and effort. The list of potential participants that must be successfully coordinated includes, but is not limited to: •

User groups/departments

•

IT/telephony department(s)

•

Carrier(s)

•

Internet Service Provider (ISP)

•

Managed Service Provider(s)

•

Management/executive sponsor

Another important consideration involves the extent to which the different entities need to be coordinated. Must an install tech from the carrier and a security escort from the client organization appear in exactly the same place at exactly the same time and stay together the entire time? The answer to this one is likely to be yes. Must the cable installer pull a pair of fiber, terminate, and test the connection before a new gateway can be installed? Most likely yes, but there is often some leeway as to how other tasks must be performed, and that flexibility must be built into the project plan.

122

Chapter 5

Plans Several inter-related plans are going to have to be put into place, desk-checked, and tested in the real world. Each of these plans will be subject to a variety of external and internal influences, as discussed throughout this chapter. As previously mentioned, internal influences are those that are, or should be, under the control of the project management team—these are easier to identify and control than external influences. For smaller organizations, there will likely be only one variation of each plan—but there should be at least one of each plan. For larger organizations, there will likely be multiple versions of each plan that will incorporate variations for a host of factors such as organization, security level, country, and so on. To be truly successful in your IP Telephony project, each variation of each plan must have its own ambassador—an ombudsman to speak on its behalf to ensure that it is getting its fair share of resources and to ensure its timely and on-budget completion. Each of the plans can be thought of as a part of the bigger organizational plan and, for that reason, all plans must be successfully completed for the overall project to be a success. Plan Validation Although it is impossible to see and avoid every problem, it is remarkably easy to identify, plan for, and avoid the major showstoppers. An important step in the creation of any plan is plan validation. I utilize five tools to help in the plan validation process. Although it is not necessary to use all tools in every circumstance, it is useful to have these tools at your disposal and become adept at knowing when and where to apply each. The tools don’t have formal names or a rigid set of rules, so I will refer to them as Visualization, Role Play, Naysayer Validation, Non-Expert Validation, and Red Teams. The tools should be applied in the order in which they are listed, and earlier tools will be applied more often than later tools in the list. Visualization, for instance, will be applied very often, informally, at the desk or sitting on the airplane, while a true Red Team is very expensive and time consuming but very often is the best way to ensure success. Visualization The first, and possibly the most valuable tool, is visualization. Visualization involves closing one’s eyes and stepping through a process visually and thinking about each step. People who use visualization extensively find that it becomes automatic—and even subconscious—and must keep a pad and pen near the bed so that they can write down things that they come up with during the night. You don’t need to sleep at your desk—visualization can be a waking exercise and forms an important part of plan validation. If you are unknowingly visualizing and coming up with useful and meaningful project tidbits, your life will be plagued with a daunting collection of those little yellow sticky notes. The value of your visualization, however, can be greatly enhanced by keeping a project notebook, or even a voice recorder, handy to be sure that all your knowledge is captured and can be put to work. The value of project personnel who actually know the practical side of the business, as opposed to simply the theoretical, is also clear. Practical experience can be increased and the value of designers and theoretical workers can be improved by assigning them to some lab and, preferably, field responsibilities as a helper or even an installer. Role Play A role play is really a visualization exercise involving two or more persons. In a role play for plan validation, for instance, visualization is performed and the steps spoken out loud, often following a plan flowchart or document. The role play is followed through until all procedures are either validated or corrected as best as it is possible to do at a desk.

123

Chapter 5

Naysayer Validation Any organization of any size has one or more naysayers—those individuals who can, and do, find fault with just about anything. Although often considered an annoyance that must be endured for purposes of intramural harmony in the organization, naysayers can represent a huge asset to the project team, as long as they are not the project manager. After doing everything possible to ensure that there are no flaws in a plan, possibly after visualization and role play to eliminate the obvious glitches, it is time to submit the plan to naysayer validation. Invite the naysayer to identify every single possible problem scenario, regardless how far-fetched, document the problem list and possible outcomes—financial loss, user hostility, lower customer satisfaction, physical injury, security exposure—and assess the potential cost of each problem and likelihood that it will occur relative to the cost to fix or avoid the problem. This is a basic risk mitigation exercise that is a crucial part of any project but is rarely ever done as a formal part of the technical project management of an IP Telephony project. Don’t have a naysayer of your own? Go out and borrow one from another department or hire an outside naysayer/consultant. The good news is that you can return them when you are done gathering their input. Non-Expert Validation A desirable next step in plan validation is non-expert validation. Non-expert validation works because non-experts do not have the same biases or frames of reference that you have relative to your project. They are in a much better position to evaluate the project from a clear, unbiased perspective. A second benefit of non-expert validation is that if your non-expert can explain a concept back to you or clearly present your idea, it must be simple enough to be understood by management and the user community and will be less likely to be misunderstood. In many cases, the non-expert might be a temporary worker or a member of the target user community. Regardless of who provides the feedback, non-expert validation offers substantial value for a very small investment of time and money. Red Teams A Red Team is an approach in which you perform a ‘dry run’ of highly complex and mission-critical presentations, plans, and proposals. This idea originated in the aerospace industry in the late 1950s and early 1960s. Whenever a project team was going to make a critical presentation or proposal to management or to their client, they would assemble an impartial Red Team from among other project teams within the company. In this way, they could get the best possible practice with the least negative exposure. The team that is being evaluated is known as the Blue Team. As project teams became less single project/single client-focused during the 1970s and 1980s, the Red Team approach all but died away and was almost entirely forgotten. That is, until the mega-projects and customer-focused teams within the telecommunications industry began to emerge in the early 1990s. The Red Team idea was revived and, although not used as widely now as it was in the 1970s and 1980s, still represents a very valuable tool for plan validation.

124

Chapter 5

Implementation Plan An overall implementation plan must be created for the project. There will be a number of subsidiary plans that must be put into place, as well, such as the training plan, test/acceptance plan, migration plan, ongoing operations plan, business/system continuity and contingency plan, and escalation procedures—all of which become part of the overall implementation plan. Training Plan One of the first questions often heard during implementation planning is, “Training for a phone?,” which is usually followed by the aside, “You pick it up, you get dial tone, and you dial.” Although user, and management, skepticism of the need for training is widespread, training is mandatory if your organization is to get the desired benefits from the new system; this is one of the elements that should be addressed in setting proper expectations early in the project. The training plan must be divided into two parts: operations training and user training. Operations training involves training on the actual guts of the system and what is needed to keep it running. Training is often under-appreciated and under-budgeted because point-and-click Graphical User Interfaces (GUIs) make a system look like anyone off the street could use them—anything but the truth. In fact, GUIs alleviate the need to learn complex and arcane commands but still require operator intelligence and knowledge to keep a system up and running. In the case that the sophisticated, and often finicky, GUI system is down, operations training will ensure that staff can use an old-fashioned dial-in modem and a command line; it is always advisable to have this skillset on staff and readily available. In situations where operations are outsourced to a carrier, ISP, or Managed Service Provider, training is still needed, but the focus is somewhat different. In a “do-it-yourself” situation, the organization must understand trouble reporting, troubleshooting, and problem resolution for upwards of 90 percent of problems they are likely to encounter. The organization’s personnel must be at least as capable in operation, upgrade, and repair of the system as the vendor or manufacturer’s field personnel. In an outsourced (or partially outsourced) situation, an organization’s staff, though smaller and more streamlined, must be experts in problem diagnosis and trouble reporting. Although there will be an initial inclination on the part of the service provider’s staff to waste a lot of time retracing the steps of the organization’s own people and redoing tests that have already been done, after the outsource organization is comfortable and confident, time will be saved and problems will be solved more quickly because the early diagnostic steps of the customer organization will be relied upon.

125

Chapter 5

Test/Acceptance Plan Based upon a combination of the list of end-user and management requirements for the new phone system and work done in the lab and in early field trials, it will be possible to put together a series of test and user acceptance plans. Regardless of the actual content, each plan, or segment of the plan must be written in such a way that a person of sufficient authority can sign off on the plan. More than one critical delivery has been botched because the person in the office at the time of the delivery said “I don’t have the authority to sign for that.” This situation must be avoided. In large, distributed organizations, one approach that works well is to have special training, possibly via a Webinar or online training session, for the individuals authorized to sign-off on the test plans. If the new voice system is going into a branch bank and the branch manager is the designated person to accept the system, an alternate person should be designated and both persons should receive training to assure that someone is trained on the acceptance criteria and procedures. Training a primary and backup person will increase the likelihood of a timely acceptance of the system and avoid delays in the schedule. The training, very simply, might include an overview of the project: why the bank is doing it and the objectives. Although overall organizational benefits are important, any training or briefing should also address benefits for the branch bank. Possible problems, fallback procedures, and contingency plans should be clearly documented, as should the test acceptance criteria, which should be as easy to verify as possible. Although a bank branch manager can be asked to ensure that a SIP Invite packet is issued to the SIP server, it is a far better idea to make “dial tone” and “ability to dial in-branch, main branch, and outside local calls” as four separate acceptance criteria. It is also possible to do remote acceptance testing, such as calling to a certain central location from the remote branch to validate a certain set of procedures. In this case, the branch manager’s role might simply be to sign off on the amount of time the tech was on site, or there may not be a role for the branch manager. Experience has shown, however, that project success is greatly increased if local personnel, especially non-technical management, are involved in acceptance. Besides being good manners, it can often create a knowledgeable on-site contact at no additional cost. Migration Plan After the design has been modified and tested prior to deployment, a migration plan must be put into place and be rolled out with discipline and precision. Rarely is a new technology rollout as visible, or subject to such across-the-board resistance, as with telephony. The migration plan is a combination of technical changes, user expectation setting, and user satisfaction exercises and must be executed flawlessly to avoid a negative impact on overall organizational effectiveness.

126

Chapter 5

Ongoing Operations Plan After the new system is implemented, and after the users have been migrated, it is time to put ongoing operational plans into place; however, the plans should be developed, refined, and validated well before they are needed. Ongoing operations plans should take into account all aspects of the life cycle of the system. For instance, assume as a starting point that the system has been properly installed and accepted, the following questions might arise: •

How will the operational status quo be maintained?

•

How will the IP phones be updated and upgraded?

•

Can new software revisions be downloaded transparently? Automatically?

•

What if a service or feature does not work? Is there a back-rev plan?

•

Are there problems in a network with mixed versions of software?

•

What about hardware refresh?

•

How often and by whose request or approval may IP phones or VoIP over WiFi devices be replaced or upgraded to newer models?

•

How is bandwidth utilization monitored and managed for applications whose usage is growing, such as video?

•

How do ongoing operations plans tie back to the Service Level Agreement (SLA)?

•

What software tools are available to automatically monitor SLA compliance?

•

When SLA breaches occur, are there software tools available to provide alerting, troubleshooting, optimization, utilization measurement and capacity planning?

•

How are SLA breaches handled within the organization’s Help desk and reported to management?

All of these topics and more must be addressed in the ongoing operations plan. Contingency and Business/System Continuity Plan A contingency plan as well as business and system continuity plans should always be a part of any project—especially one of the visibility and potential impact on an organization as a IP Telephony project. Having these plans in place, even if they are never put into play, will impress management and users with the scope of planning and give them confidence that all possibilities have been considered. In addition, the plans will be available when and if needed. Periodic drills and reviews should be performed to ensure that the plans will work if put into action.

127

Chapter 5 Contingency Plan Retreat, though not a savory choice, must always be kept as a viable option. A well-planned retreat has saved many a project and rescued the very valuable “political capital” needed for credibility within an organization. A contingency plan for terrestrial wired telephony might be as simple as using cell phones as a fallback if the new IP phones aren’t working, but the procedures should be clearly spelled out, including details such as access to voicemail during the transition—from the cell phones if needed. A fall-back plan for IP Telephony should also include a lab-tested and field-verified approach for recovering the system to the last-known good configuration as well as a timetable for making a go/no-go decision for various milestones within the rollout. If, for instance, a system is to be operational at 8:00 am on Monday morning and the rollback procedure, conservatively, takes 2 hours to execute, a go/no-go decision point should be scheduled no later than 4:00 am on Monday morning if the new or upgraded system has not yet passed acceptance testing. Note that the 4:00 am decision point is actually 4 hours, not 2 hours, earlier than the 8:00 am operational time. The extra time allows for some assessment to be done at 4:00 am and extra time in case the estimated time for the rollback is off, and to validate proper operation after the rollback procedure. In any case, an operational phone system is online at 8:00 am.

Business/System Continuity Plan Overall business and system continuity plans are inextricably interwoven with the business and system continuity plans for specific systems or subsystems. An organization, for instance, may have a plan, and hardware, in place to ensure that a building has backup electrical power via a generator, for instance, in case of loss of primary power. This alleviates the IP Telephony project team from responsibility for purchasing and maintaining such a system, but this does not alleviate responsibility for ensuring access to such a system and for participation in periodic testing. As an example, power outlets that have Uninterruptible Power Supplies (UPS) and/or generator backup are often orange. In this example, IP Telephony project procedures should ensure that all IP phones, or equipment racks providing POE, are plugged into the orange outlets. Beyond that, procedures should ensure that no additional, non-essential equipment, such as personal radios or space heaters, are plugged into the designated outlets. The IP Telephony project team must also ensure that the backup power system can accommodate the load put on the system by their equipment and participate in periodic testing. The Value of Reliability What is the actual cost of an outage or other problem, per event, regardless of duration? What is the actual cost related to the length of the outage or problem? Are there thresholds at which the cost rises? And, what are the other impacts, both actual and intangible, that result from an outage? Every outage, regardless of duration, has a cost simply because it happened. The cost may be employee confidence in the system, customers who are lost because they switch to a new service, or stock prices being negatively influenced by accompanying bad press. In addition to one-time costs, the cost may accumulate over time. As an example, having to pay overtime to get time-sensitive transactions entered after a computer network is back “on the air” or having to pay overtime, or miss the window of opportunity, for making telemarketing calls to homeowners during the early evening hours. Or, as is often the case in financial transactions, mind-boggling sums might be lost if a transaction cannot be performed. Numbers into the millions of dollars per minute are not uncommon for certain financial systems. Only by knowing the true cost of a service-impacting outage can you assess the true cost and true value of reliability.

128

Chapter 5 Escalation Procedure To ensure that escalation is applied in a positive way to the IP Telephony project, you must be certain that escalation timetables are clearly documented, locked in, formal, and automatic. That is to say that the Tech I who is onsite installing the soft switch knows that if he or she is not done within 4 hours of starting, or by 6:00 pm, or whatever, they are to immediately call and notify the Tech Manager and to get reinforcements. It is also incumbent upon the organization to ensure that the reinforcements are available and are aware of their trigger points for automatic escalation. An underlying consideration is to set a bigger penalty for not performing an escalation than for performing one, and that the blame game be played only after numerous similar failures on the part of the individual. And, even then, only in a positive manner aimed at solving the shortcoming, even though the solution might involve training, retraining, reassignment, or termination.

Internal Issues Now that you have a hint as to the range of potentially project-impacting issues over which you will have limited, or no, control, let’s turn attention to those items over which you should be able to exert more control. We can then dive into our case study for a greater appreciation of how these elements can impact an actual implementation. Creation of Baseline and User Profiles During the process of determining the needs and objectives of management and users, a set of criteria emerged that were the basis for system development and validation and, most likely, have been incorporated at this point into the SLA. From those requirements, additional input from management, and a review of any applicable security and/or acceptable use policies, a baseline user profile and specific profiles for classes of users must be developed. The specifics will differ based upon your exact system but will include a range of capabilities: •

Permission to use outbound long-distance and international calling

•

Applications that are accessible from different platforms

•

Amount of system memory and processor capability

•

Number of calls that may be stored in voicemail

•

Delivery of voicemail via email

•

Whether voicemails should be retained on the server

129

Chapter 5

Bringing All Users to Baseline After the organizational, or departmental, baselines have been established, it is important to upgrade any users who do not meet the baseline profile and train them to be sure they are getting the most benefits from the new service. In some instances, system users may have capabilities or administrative rights that they will not retain when moving into the new system. The housecleaning aspect of a move to a new system is important, as users and their privileges can morph over time, usually gaining capabilities rather than losing them. One thing that should not be done is to migrate users wholesale while maintaining users’ old rights and privileges. For a variety of reasons, mostly operational and security related, there should be a minimum number of possible user profiles and very, very few exceptions. The politics may be tricky and the battles might go to the highest levels, which is why a policy of minimum number of user profiles with users receiving the minimum number of rights that will allow them to accomplish their jobs, should be established from the very beginning. Additional Capabilities by User Profile Once baselines are established, additional user capabilities and permissions may be granted for specific user profiles. A common user profile that meets the needs of a certain class of user is always preferable to personal profiles that must be maintained and administered resulting in a nightmarish and unmanageable maize of profiles and permissions. Security Issues We must also consider the security of our system and not forget that attackers will find combinations of capabilities and permissions that give them access and allow them to compromise the system. For this reason, profiles should be established and reviewed before being implemented and should be audited and reviewed periodically. Security of IP Telephony systems is particularly difficult because IP Telephony systems are susceptible to all of the security vulnerabilities of the systems upon which they run as well as new vulnerabilities related to IP Telephony generally and manufacturer’s implementations specifically. For instance, a Denial of Service (DoS) attack that exploits the SYN exchange that begins each TCP session and is aimed at a server running IP Telephony is not a IP Telephony security problem, per se, but will, albeit indirectly put IP Telephony out of service. On the other hand, IP Telephony may be attacked directly, for instance, by exploiting weaknesses in the way that Uniform Resource Identifiers are parsed, which will compromise the IP Telephony service but will not impact underlying protocol layers or other services running on the same server. Security is so important that it should be carefully considered at each step of planning and deployment in conjunction with the internal security department and, as needed, outside security professionals and consultants.

130

Chapter 5

Liaison with Security Department It is important early-on to establish a liaison with the internal and/or service provider security departments, to seek their input, guidance, and collaboration and to keep them aware of what you are doing. Very often, folks from the security department have as much knowledge about your needs and applications as you do about their responsibilities. For this reason, it is a good idea to start any meetings with a briefing about what you are trying to accomplish and why and to request and expect a similar briefing from the security people. When working with the security people, you can increase the likelihood of success (that is, that your efforts will not be blocked or thwarted in any meaningful way) by gently reminding the security folks charged with your protection that they are in the role of providing a valuable service to you and that you are the customer, not the other way around. In many cases, the very best security people have spent at least a part of their careers in situations in which running the show has had real and serious consequences in terms of financial loss or national security. It is a bit different in the commercial world, but the security viewpoint should always be considered and, in the case of a tie, a debate can be held to allow an upper-level manager to make a policy decision. Firewalls and Proxy Server Issues Firewalls, proxy servers, intrusion detection systems (IDSs), and similar tools are the stock-intrade of the security business, and the value of properly implemented security systems cannot be overstated. However, two issues come into play when discussing security as it relates to your IP Telephony project. The first issue is improperly implemented systems that degrade, disrupt, or outright stop VoIP services. The second issue is with properly implemented systems that do not allow VoIP. In the case of firewalls, for instance, most firewalls today do not make simple “pass/no pass” decisions based upon simple criteria such as source and destination IP addresses. Today’s firewalls often perform sophisticated analyses called heuristics, which protect against as-yetunknown problems, or stateful inspection, which does sophisticated analysis of the interplay of multiple levels, or layers, of protocols. In the best case, this process only adds some delay, but in the worst case, these systems might cause sessions to be dropped or blocked entirely. Proxy servers are another type of system that can often get in the way of VoIP. A proxy server acts on behalf of a user. To protect a user from attack and maintain anonymity, a user’s browser, for instance, might make a request of a proxy server to do some action on its behalf, to retrieve a Web page, for instance. In this case, the user connects to the proxy server, and the proxy server connects to the Web server. Each of the Web sessions requires additional resources on the proxy server, and most proxy servers are sized and configured based upon the number of simultaneous sessions. If, in your new VoIP service, VoIP calls were to replace browser sessions on a 1-to-1 basis, you would probably be OK, proxy-server wise. But it is likely that the sum of simultaneous browser and VoIP sessions will be greater than browser sessions only before implementation of the new system; thus, you run the risk, especially during the busiest times, of overloading the proxy servers. Security systems—be they firewalls, proxy servers, or IDSs— should always be a part of system testing.

131

Chapter 5 IP Addresses and Port Numbers Most prevalent applications on IP networks, such as browsers, file transfer, and email, use wellknown port numbers to communicate and, therefore, may be readily identified by firewalls and other security systems. IDSs and stateful inspection services can also determine fairly easily whether the behaviors of these applications are proper. A problem with VoIP is that it does not use well-known port numbers, but rather, registered port numbers, and requires special configurations on firewalls and other security systems to handle different “flavors” of VoIP. By different flavors, I mean VoIP originated from different manufacturers. This is especially problematic in environments in which VoIP from multiple manufacturers might be employed, such as an environment where employees choose the VoIP system they will use, or a system that is shared with clients. This is far less problematic in a controlled environment where the organization controls all the parameters, but it is possible to have both situations. IP Addresses, Port Numbers, Sockets, and VoIP IP addresses plus Layer 4 (TCP/UDP) port numbers equal socket numbers. Sockets uniquely identify the endpoint of a connection in the network at a moment in time. If you think about a street addressing analogy, IP addresses get you to the building (that is, server or other IP host, such as a user PC), and the port number actually gets you to the office in the building (the application). All port numbers fall into one of three categories: well-known, registered, or ethereal, which means temporary. Most widely used applications have well-known port numbers, which makes life easier for administrative and security use. If a widely used application is authorized for the system, its well-known port identified by its well-known port number is either left “open” or opened provisionally for information packets to flow through. If a given application is not authorized, or not authorized for the IP addresses, or some other criteria, then it is closed and any attempt to go through that port is seen as a potential security violation. The problem with VoIP is that it was not originally included in the list of common, prevalent applications and, therefore, does not have a well-known port number. It is too late to retrofit it into IPv4, the current version of IP being used most widely in the US today. VoIP applications, therefore, must use registered port numbers, which vary by manufacturer, protocol, and software publisher.

Consider, for instance, a situation in which a large hotel chain provides an IP network that is used by both guests and employees. In this case, the IP network should be segregated for security and performance reasons because employees will use the carefully chosen and implemented VoIP solution internally and hotel guests will use anything they want in an ad hoc fashion and may, or may not, get the voice quality they need in the open, “best effort” world of the dynamically shared private intranet/public Internet connection.

132

Chapter 5

NAT and Internal and External IP Addresses IP addresses come in two basic flavors—registered and unregistered. The registration process ensures that no two user organizations will have the same IP addresses on the Internet because the registration process provides unique sets, or blocks, of IP addresses. This does not mean that organizations cannot use their own IP address assignments on their own, private IP networks: they can. In addition, there is not a problem with this situation as long as the private IP network, with its own set of IP addresses, is not connected to the Internet. If it is desirable to connect the two networks with conflicting IP addresses, this must be accomplished using a device, such as a router, that performs a special translation function of the internal, potentially duplicate, IP addresses to external, unique IP addresses. This function is called Network Address Translation (NAT). In many cases, internal users do not need access to the Internet, and there are many more internal users than those crossing the NAT boundary. The importance of NAT is clear, but the reason why it deserves special consideration in VoIP systems is that the use of NAT often causes a disconnect in the mapping of the underlying IP address to the higher-level phone number or User Resource Identifier (URI) used by VoIP. The solution to the NAT problem lies in either ensuring that all “internal” IP addresses are registered, or using a separate IP network internally for VoIP. Assuring that all addresses are registered eliminates NAT, which is not practical for many organizations; Using a separate IP network internally for VoIP simplifies the issues of managing VoIP but raises costs and network complexity and completely kills the multimedia, triple play network concept. It is also possible to place servers, such as VoIP or SIP proxy servers, strategically on the Internet/registered side of the NAT function so that remapping is done after the relationship between the IP address and phone number or URI is not crucial. Another approach is to “bury” the IP address relationships within lower-layer transport systems, such as Virtual LANS (VLANS) or Virtual Private Networks (VPNs). Internal and External Phone Number Mapping The foregoing discussion of the use of internal IP addresses that were assigned without reference to standards or registered assignment and the potential to encounter duplicate numbers is not limited to IP addresses. Many organizations, especially the world’s largest, have the same issue with traditional telephone numbers. The problem is solved in much the same way as IP addresses, where internal phone numbers are used internally but are mapped to external numbers either through PBXs, IP PBXs, or some other gateway function, on their way into the public switched telephone network (PSTN). This is a large issue and should be part of a serious discussion prior to implementation.

133

Chapter 5

Flash Cut vs. Parallel Operation The next consideration of migration is whether a system should be flash cut—always, of course, with a rollback plan in place to return to prior operational levels should a disaster of some type occur during the cutover—or allowed to operate in parallel for some period of time. The flash cut can be a bit of a bold move and disconcerting for the end users, but operationally, it is usually a lot cleaner than having two systems in place for any period of time. A happy middle ground is to do a phased operation, possibly in two or three steps, in which an individual phone user is flash cut, gaining their new IP-based phone and losing their old phone all in one fluid motion, but there will still be old phones in the office area for some period of time until the final set of users are cut over. If users are properly prepared and trained, this type of phased implementation can provide a psychological safety net for the users but keep costs and the complexity of the interim network operations under control. Migration Priority and Order The order in which migration will occur and the priority of users within the list should be the cause of much debate. Invariably, management will either want the highest ranking people to go first because they are the most important and should get these shiny new tools before the masses, or they will want the highest ranking people to go last because any problems in implementing the new technology will have the biggest negative impact on them. There is rarely a middle ground. What might make more sense is to migrate users who have been involved in system selection and implementation in the first wave. These “friendlies” are more likely to provide insightful commentary and useful observations in the early phases as opposed to creating political firefights that must be fought before migration progresses. The earliest migrations should be as close geographically to the source of technical support as possible and, furthermore, capable of receiving overnight packages. This may mean the headquarters office, but it does not mean that the CEO’s office should be migrated first. Involvement of End Users in Acceptance The importance of involvement of the end users in the implementation and migration of the IP Telephony system has been stressed throughout the project. It is no more or less important at the implementation and migration phase. Continue to work with the end-user community, generate good “word of mouth,” work to address all issues quickly and thoroughly, and be aware that the performance of your network very well might begin to change as more and more packet telephony users are added to the network. For this reason, keep your feelers out in the user community, get early insight into possible problems, and use your friendlies to detect problems early, before they are fatal.

134

Chapter 5

IP Telephony Implementation and Migration Case Study At this point, you know a lot. When you combine the previous four chapters with this chapter and your own broad personal knowledge gained from reading, learning, and, quite possibly, your own experiences at designing, implementing, and using packet voice, it adds up to a lot of knowledge. The question becomes: “How can you customize your knowledge to your specific needs to allow you to directly apply what you know to ensure the success of your IP Telephony initiative?”. The reality is that level of customization requires a lot of interaction and collaboration that is not possible in the current medium. So, let’s do the next best thing—explore how the many hundreds of different factors that have already been discussed were combined to create an implementation and migration architecture—a blueprint—for a global company. The hope is that you will see patterns and approaches that will be helpful to you and that in this way, you can start down the road of applying the theoretical to the practical. Hopefully, this case study will give you more insight and food for thought. To fully understand the case study, take a look at the Request for Proposal (RFP) used by the company that is the subject of this case study. You might even find the document useful as a starting point from which you may add, change, and delete to create your own RFP. A generalized version of the RFP, which has been modified to be more general as well as to hide the identity of the company is available from at http://nexus.realtimepublishers.com/content/DGSDVIP_RFP.doc.

The following case study will have many elements that can be applied to you, or at least, using your imagination, you can put yourself in the action. If you are not a global giant, you might not need to consider the legal and regulatory environment in many countries—simply in your own country. If you are a carrier or service provider, maybe you can forego the selection of the transport network and use your own, or maybe you should consider using the transport services of another company for your off-net calls. Maybe you don’t want to manage your old gear as a part of your plan; then don’t, but at least take a look and see how things came together in this real-life scenario. The Three Tracks There is an oft-told story about three blind people and their description of an elephant. The story goes something like this: Three blind people go to the zoo and are given the opportunity to touch an elephant. Having never seen an elephant, as all of them were blind from birth, they didn’t really know what to expect, but having felt many other things, they thought they did have a basis for their observations of the elephant. After they had all had a chance to touch the elephant, they were asked to describe the beast. One person, having felt the trunk, said that an elephant was much like a snake, only much bigger. The second person, having grasped the tail, described the elephant as being like a huge broom, swishing this way and that. The third impression was as a result of handling the elephant’s leg; the third person described an elephant as being much like a large tree. The reason for pausing at this point to share this story is that it illustrates a common situation with IP Telephony projects—although most folks have never encountered this IP Telephony animal before, they all bring prior conceptions, often misconceptions, about it, and they all tend to view it from their own perspective. The participants in your project will often see what they see based upon their experience and job status. It will be the job of the project manager to make this specialization work for you, as was the case in the following case study.

135

Chapter 5 Three different groups are involved in the implementation of the global IP Telephony project: they are the business group, the technical group, and the legal and regulatory group. The business group sees the business benefits and pitfalls of the global IP Telephony project; they confirm the Return on Investment (ROI) calculation and the tax ramifications of equipment depreciation and worry about factors such as Moore’s Law. The technical group is concerned about the protocols, devices, and deployment. They concern themselves with considerations such as Quality of Service (QoS) in a multimedia network, network security, and survivability of systems in often harsh field conditions. The legal department is worried about all things legal and regulatory: contracts, local regulation, corporate governance, and similar worries. Instead of re-educating everyone in new ways of doing things, the decision was made early-on to harness the natural order of things and design the project blueprint around three tracks: business, technical, and legal/regulatory, and to closely manage the points of interaction between the tracks. Although it is the job of the overall project managers to see the 50,000-foot view, the workers are best focused on their own area of specialty. Let’s explore the project based upon these three tracks as identified in Figure 5.1.

Figure 5.1: IP Telephony project roadmap.

136

Chapter 5 A kickoff meeting was held with representatives of all groups present to allow project participants to meet each other and to get a sense of the “big picture,” which was, literally, a big picture. Each group received a master E-size copy of their organization’s master project map (see Figure 5.1), measuring nearly 3 feet by 4 feet. It was agreed that the master blueprint would remain unchanged, but it was the job of each track to define the specific, detailed, approach for each of the boxes represented on the master project map. The project map, and all subsequent detailed drawings from the tracks, are done in Visio and each track was allowed to use any other tools of their own choice for managing their part of the project, but Visio was the common way of representing project relationships, eventually yielding several hundred detailed diagrams. Business The initial job of the business team was to assess management’s and user’s needs and desires for the new system—both in terms of actual functionality and business impact. In many organizations, this task would have been relegated to the technical team, but management in this case did not want their new telephony initiative to be based upon what was technically possible but rather wanted to apply available and immediately emerging technology to specific business needs and business outcomes. The management and user capabilities audit determined all current telephony capabilities that were mandatory and those that were desirable in the next-generation telephony project. Audits included user surveys, Postal Telephone & Telegraph (PTT), carrier and service provider record audits and billing reviews, and switch and PBX audits. One noteworthy outcome of the audit process was a realization of how much attrition of expertise about the older systems there had been. In some cases, retired employees were brought back as consultants to gather configuration details and call detail record information from older vintage switches and Private Branch Exchanges (PBXs)—functions rarely performed on some of the older gear in the network. This realization, in and of its self, was a wake up call for management. ROI and TCO An important consideration in the management review was the financial aspects of the project. Although the details are confidential, it is possible to share some broad observations. For instance, since the mid-1990s, the company had been realizing approximately 20 percent yearover-year savings in their voice and data networking areas with increasing costs in their telemetry and video areas. The savings represents two factors: the first factor is riding an aging investment in voice and data infrastructure on the back side of its depreciation curve and the second factor is demanding cost concessions from vendors and service providers until there was almost no margin left. It was clear that the new IP Telephony initiative was going to put voice and data infrastructure into the increasing cost area, at least for the foreseeable future, and that vendors and service providers could give little more in discounts.

137

Chapter 5 In the first case, all benefits of the new network needed to be clearly identified to upper management and metrics devised to show progress. This was accomplished by a senior telecom and networking management taskforce who approached the task as if their survival depended upon it—and it did. Spending more without producing tangible results was out of the question. In fact, the senior vice president in charge of the taskforce commented that “there is no such thing as strategic in our world: it is purely tactical. Tactical means that you can show hard cost savings or other benefits and strategic means that you can’t. We have become purely tactical.” The second area was addressed by taking a different approach to vendors and service providers. The relationships were redefined to foster a more collaborative engagement, which allowed the company to get more value from a relationship with fewer key vendors, carriers, and service providers, and to pay a bit more money to each. This approach also leveraged the relationships to reduce headcount, especially in very remote areas, in what might be called a very closely managed near-outsourcing arrangement. In terms of actual ROI and Total Cost of Ownership (TCO) targets, the ROI and TCO in this case study represent nearly unrealistic goals for most companies. Most companies cannot purchase products and services on the scale of the case study company. One of the reasons that scale matters is that very high costs in certain areas can be averaged out by lower costs in other areas. This averaging effect is easier with tens of thousands of users. On the other side of the coin, however, even slightly higher across the board costs will soon spiral out of control if not contained. The typical ROI for the company is 12 months, but because of the scale of the changes that needed to occur, the ROI was calculated at nearly 14 months—considered good by many organizations. TCO, not including incidental expenses incurred on an individual basis such as Internet access service charges at hotels for traveling employees, is around $38/person/month for the VoIP portion of the network only. The network includes other IP Telephony technologies, such as Voice over ATM and Voice over Frame Relay, with higher TCOs. The company is migrating away from aging traditional and non-VoIP systems as quickly as possible, convinced of the financial and operational benefits of a single IP infrastructure company-wide. Establish Site Profiles The broad user and management capabilities audits were combined for management review and the result was a master set of capabilities and target business objectives that were compiled into MS Excel spreadsheets. From the spreadsheets, user and management representatives from each of several thousand sites and locations, such as small office/home office and mobile workers, were able to identify their specific needs. The spreadsheet results were used to develop a minimum number of profiles. One of the inputs to the process of developing the client site profiles was the client country legal/regulatory profile. For instance, in certain countries in which the company operates, it was against regulations, or in some cases flat-out illegal, to do VoIP. In other cases VoIP is allowed, but there are restrictions on encryption of information, which for this company, is a very desirable aspect of VoIP that will keep information more secure than over traditional telephone lines. Although this step will not be needed for many IP Telephony projects, it is at least worthy of consideration. Output from the site profile creation was used as input to the development of private network, IP VPN, and completely outsourced Layer 2 VPN strategies in the technical track. 138

Chapter 5 Rough-Order-of-Magnitude Pricing The next step in the business track was to take configuration inputs, bandwidth requirements, and maintenance and support information from the technical track for three different network strategies: a completely private multimedia IP network, with the company basically acting as its own internal carrier; an IP VPN strategy, which completely outsourced the IP VPN with very strict SLA requirements; and a completely outsourced global Layer 2 VPN network, with similarly strict SLA requirements. Comprehensive cost and pricing models were developed for each option and compared with target ROI and TCO objectives. Management Review and Site Prioritization Various scenarios were presented to management based upon which areas of the company, and even specific offices and locations, would achieve the greatest benefits from a new telephony system. The final result of this step is that “green field” locations, that is to say new business locations coming online would, in most situations, be the first recipients of the new system, if a business need existed. If a clear business need did not exist, the green field sites would receive older telephony equipment that was displaced from existing sites that were upgraded. Older equipment displaced as a result of a migration would be returned to a surplus equipment inventory and would be used to extend the life of offices that had older telephony gear in place and did not need or did not yet need the new packet voice system or were in a country with legal or regulatory restrictions on the use of VoIP. The surplus equipment management process will be described in more depth shortly. Initial Migration/Green Field Plan and Renegotiation of Favorable Contracts The next step was to translate management’s needs into an actionable plan for the sites, order of migration, and action identified by management. This process resulted in the initial migration and green field plans with green field opportunities taking precedence over migration where there was a conflict for resources. Armed with actual requirements and funding approval for the new sites and a clear indication of the needs of existing sites for the new systems, vendors and bandwidth providers were engaged in a renegotiation of existing contracts or negotiation of new contracts for green field sites. This process was part of a feedback loop that allowed management to reprioritize based upon favorable or unfavorable outcomes from the renegotiation process. Joint Migration/Green Field Planning Effort With a clear set of tasks, based upon business imperatives and management direction, it was now time for a joint meeting between the business team and the technical team to determine specific actions, dates, times, and responsibilities. The technical team had prepared templates for the actions needed for each site profile and could combine them, clear up any discrepancies, and create a task list, time line, and other project management tools, and integrate the operation into the overall schedule.

139

Chapter 5 Contracts and Delivery Contracts and delivery, although processed by the legal team, were managed jointly by the technical team and business team and were a key output from the joint migration/green field planning effort. Contract specifications had to align with key project dates and delivery of goods and turn ups of services had to be identified and provided back to the legal team to go to purchasing fulfillment and accounts payable. Although these details are often not treated seriously or taken into account, there are possible operational circumstances; consider, for instance, a location whose ISP contract is not paid and they lose their service, or the impact on corporate profitability if fixed assets are not received and accounted for properly. Although seemingly financial functions of little interest to the technical team, the reverse is actually true: they are of the greatest importance and should be taken into account at each step of the project. One of the outcomes of the business track was the surplus equipment management effort. Inventory and Fixed Asset Accounting and Disposition of Old Equipment One of the primary reasons for a migration to a next generation of telephony system—any nextgeneration system—for many of the company sites had nothing to do with advanced new capabilities. The real reason was manufacturer discontinuation of older systems and models and the diminishing availability and rising cost of components for maintenance, repair, and expansion of their existing, older vintage, telephony systems. A solution to this that extended the lifetime and created bottom line benefits of the older systems was to take equipment that had been decommissioned during the implementation of the new system and send it to one of three regional warehouses for refurbishing and placement in service, either for repair or upgrades, in other locations that were still using the older equipment. This approach took what might otherwise have constituted garbage and repurposed it in a unique way that helped the company achieve its overall business objectives. The inventory management and fixed asset accounting for each of the components moving into and out of the regional refurbishment and warehouse centers was managed in the same way as all other assets of the company and will be until their ultimate disposal. Implementation of Changes to Business Operations There are other important considerations in the business track that are noteworthy and interesting that appear in subsequent detailed diagrams but that are not specifically enumerated in the project map. They are: •

Training

•

Dealing with user reluctance

•

Standardization

•

Impact assessment

•

Changes to processes

•

Staff implications

140

Chapter 5 Training of users was accomplished via a multi-tiered approach. Representatives from each office, plant, remote location, department, and so on were trained on the system operation and any special capabilities specific to their business function. These representative were the same people who had been involved in the needs assessment, design and implementation/migration planning all along—being, in effect, ambassadors for the new system and, therefore, duly enthusiastic about its success. The representatives trained other users in both formal and informal settings and were available as liaisons to customer support, creating a frontline of support that was able to answer many of the basic questions and avoid too big an additional demand on the Help desk. There was, of course, initial user reluctance, in many cases being somewhat strenuous. Most users eventually became familiar with the new systems, though many never completely got over their anxiety and were likely to blame the new phone system for any mishap. Standardization of business systems and practices is somewhat different than the standardization discussed in the technical track. In the case of standardization of business practices, use of a new, single, consistent IP Telephony system organization-wide, regardless of the individual features or enhancements for a given operational area, allowed for a simplification of work processes, better transportability of employees between offices and job functions, and an overall organization-wide focus on a single system—all of which resulted in both tangible productivity improvements and intangible user satisfaction with the system. Impact assessment was accomplished via a series of formal user surveys, getting higher marks from users over time, and productivity tools capable of measuring everything from “time to dial tone” and “connect time” and the overall impact of the project was judged to be positive and in most cases met management objectives or changes were put in place to get the system in line via changes to processes. One last area of concern, and an area where management was upset somewhat in their initial thinking, was that of the staff implications of a move to a IP Telephony system. In management’s initial view, after the migration was complete, it would be possible to lay off the entire old telephony staff and translate the savings in salaries and costs directly to the bottom line. Their strong belief in this potential benefit was rooted in the belief that “VoIP is just another IP application. Get some IP phones, plug them into your existing IP network, and you’re ‘good to go.” Several things conspired to frustrate them in these efforts, not the least of which that this belief is absolutely false—as previous chapters have ably demonstrated. Management was warned in very clear terms to avoid this situation. What management actually found is that they were replacing individual telephone, data, and video engineers with more valuable, less available multimedia “triple play” engineers. This replacement process was often through hiring but, more often, through training. The company was able to let the lowest-tier of least productive employees in each category go and to keep the new breed of triple play engineers. Much of the training was formal from outside sources, but some of the training was internal and on the job. The lesson in this for organizations implementing IP Telephony is to never neglect the value of the body of knowledge that traditional telco people bring to the project. The transport has shifted to IP, but the basic application—humans speaking with each other—remains unchanged.

141

Chapter 5 Technical The technical track runs in parallel with, and occasionally overlaps, the business and legal/regulatory tracks. It begins with a review of standards’ status and maturity. Review of Standards’ Status and Maturity At the time of the case study, the review of standards’ status and maturity was a much more important element than it is today. Today, for instance, systems based upon the Session Initiation Protocol (SIP) are a given, while at the time of the case study, H.323, SIP, and ETSI TIPHONbased systems were all being considered. Standards, and even Internet Requests for Comment (RFCs) have come a long way both in terms of their refinement and the way in which they will be implemented in the marketplace. Such a review as this today will start by identifying the underlying assumptions, or givens, and then delve into what may be less accurately known; this analysis will often go into the lower layers of the network, as well. Only passing consideration was given to Multi-Protocol Label Switching (MPLS), for instance, in this case study, while Ethernet MANs/WANs, MPLS, Virtual Private LAN Service (VPLS), and Layer 2 Pseudo-wire VPNs are all options for the type of QoS-assured networks needed for success in packet voice generally and VoIP specifically. Develop Private Net, IP VPN, and Carrier/PTT Strategies Following adoption of a set of standards, separate strategies were developed for using a completely private network, an outsourced IP VPN based on MPLS, and a Layer 2 carrier/PTTprovided VPN. What eventually occurred was the development of a compromise plan for a hybrid network that leveraged the best aspects of one or all of the approaches depending upon a fairly strict set of selection guidelines developed jointly by the business team to ensure that business requirements such as ROI, TCO, availability, and reliability were met; the technical team to ensure that technical elements were properly addressed; and the legal/regulatory team to ensure that a specific set of characteristics was contractually possible. For instance, some locations, especially in areas in which the company has a strong presence, are sufficiently dense to be serviced by a company-provided private IP network; in some very remote locations, the company operates their private IP network over PTT-provided leased lines or over global carrier Ethernet WANs. In other areas, sites are served by global IP VPN connections over a service-provider MPLS cloud. The actual selection criteria were put into an Excel spreadsheet to allow a multi-column menu approach in selecting what to provide to a given site based upon a variety of criteria, including application mix, QoS criteria, total site bandwidth, location, traditional vs. IP telephony, and other similar qualifiers.

142

Chapter 5

Private Backbone Design/Upsize for Voice Traffic and Backbone Re-engineering As an outcome of the design process, the company instigated the steps required to prepare their existing private backbone to handle both the new voice traffic that would be originating and terminating on their network as well as transit traffic that would be crossing their backbone. The company used a set of network assessment tools in this phase that were invaluable in gathering live data about the operation of the existing network and predicting the possible outcomes, and resulting impact on, VoIP voice quality of certain design changes. The company found that the outcomes were not always what they thought they would be and certain, often less expensive, architectural and operational changes yielded better overall improvements in forecasted voice quality than more expensive changes. Establish Common Standards and Test Plan After a careful evaluation of the three architectural choices, and the adoption of a hybrid, best of breed approach, a set of standards and test plans for the voice application were developed. By establishing a set of standards and testing methodologies for the overriding voice application, as opposed to basing testing on the underlying technologies, the company came closer to measurements, testing, and problem resolution techniques, that would have a more direct impact on their users than they would have otherwise. Delay, for instance, was calculated “mouth to ear” as opposed to from network edge to network edge or site to site. Mean Opinion Scores (MOSs) were estimated and R-Values calculated for purposes of comparison, and SLA compliance and a single, comprehensive, set of metrics was developed regardless of where in the world the voice system user was connected. Establish Private Net, IP VPN, and Carrier/PTT Demo/Test Capabilities To fully test functionality and validate the network design, full test-bed networks were created to match the private net, IP VPN, and Carrier/PTT systems. These test beds also allowed hybrid configurations to be developed quickly for purposes of hybrid testing of real-world configurations. Engineers, both from the internal company staff and from outside sources including service providers, carriers, PTT authorities and consultants who had worked on the test systems, were designated to either the support team or migration team. Support team members would support the migration and ongoing operations efforts and would be given a permanent staff position. Members of the migration team were temporary and would not remain on staff after the migration was completed. This distinction was not completely enforced, however, and some support team members left or were terminated and some migration team members were given permanent positions, based upon their performance during the migration and other factors. Training/Staging for Migration Teams Most of the training for the migration teams occurred in the actual staging of equipment that would be taken to the field. Systems were installed and initial testing and configuration performed before the systems were taken or shipped to the field. Migration team members received a minimum of training because an investment in their training would not yield a return in the long run and they would be supported centrally by support team members.

143

Chapter 5 Training for Support Teams Because the support teams were to be the ongoing multimedia support for the network, the interim period prior to and in the beginning phases of implementation were a period of nearconstant formal and informal training. The support team members were to be the elite engineers, and a substantial investment was made in their training and at each step along the way the specific training they received was tracked back to the business benefits of the skills and a sort of informal Return on Training Investment (ROTI) was calculated. Review of Planning and New Telephony Project Rollouts After a review of planning, and appropriate modifications for specific circumstances, the new telephony project rollout was begun. If your network is to go no further than the borders of the contiguous 48 states, your legal and regulatory issues will be minimal. If you use a service provider, rather than taking a do-it-yourself approach, the issues will be even less complicated. However, if you will be operating in multiple countries, using a variety of providers or some other complicating factors, you will be spending more time on legal and regulatory areas. At the very least, use the following items from the case study as a checklist to determine whether you need to be concerned about the specific areas. Legal The legal track is executed primarily by attorney’s and includes contracts, compliance with specific legal requirements, and the resolution of disputes with areas outside the company or legal complaints arising within the company related to human resources issues. Closing Contracts for Old Telephony System One of the most important legal areas, from a standpoint of financial impact, is in closing out contracts for the old telephony systems and related services. Ordinarily, audits of bills are conducted within the business track with input and guidance from the technical track and, to the extent that any moves, adds, or changes comply with the existing contracts can all be handled without legal intervention. The reason the legal department is involved in this area, more-so than in other situations, is that the changes to contracts, and contract cancellations, related to the new packet voice project do not always track the dates and conditions foreseen in contracts. It is also often the case that contracts can be renegotiated, often at more favorable rates or conditions, as a part of the project. New Contracts In addition to disposition of existing contracts, new contracts to replace old contracts or with completely new vendors need to be negotiated. In this case, the legal team collaborates with the business and technical teams to ensure that the contracted goods or services meet the stated business needs and that any accompanying SLA is properly constructed and administered.

144

Chapter 5

Regulatory Regulatory issues within the U.S. could be the topic of another guide entirely; the good news is that regulatory issues generally do not directly impact private organizations that are constructing their own private packet voice systems—though they will impact the companies providing goods or services. This is not the case in many other countries around the globe. Compliance with National Law and Telco/PTT Regulation Any company implementing a packet voice capability must be aware of local regulation and law governing voice systems. Some countries are very open on the topic while others prop up corrupt regimes, or those seen as corrupt by the U.S. government, using the hard currencies they receive in settlement compensation when the money collected for calls to their country exceeds the money collected for calls from their country. In other situations, the issues might be less regulatory and more contractual, where a contract for voice services through a carrier or PTT authority includes minimum minutes or dollar volumes. In the case of the case study subject company, a database was created with all these factors and more, and that database was used as an input into the business and technical tracks. Some of the other areas tracked in the database are highlighted in the following sections. Security and Privacy Issues

The extent to which voice communications, and often data communications, can be made private is often strictly controlled in certain countries or under certain conditions. Although it is very desirable for certain organizations to use sophisticated encryption techniques to keep their information secure from competitors, governments, and even non-authorized employees, the tools to do so are often prohibited. This is a problem in general for fixed location operations, but for the mobile employee who may be traveling from country to country, knowledge of these restrictions is very important as possession of certain hardware and software may be cause for detention at the borders or by law enforcement authorities. Carrier/PTT IP Bypass and Local Tail Drop-Off

One of the cost saving elements of VoIP is that it allows bypass of the traditional carrier/PTT systems, allowing those older systems to be less heavily loaded and eventually removed. This capability is not allowed in many global jurisdictions nor is local tail drop-off, which allows delivery of a local call on a traditional circuit that has originated elsewhere in the network. This restriction is often related to calls carried between company locations for individuals outside the company but can restrict intra-company systems use as well. PBX and Gateway Issues

Many of the restrictions on VoIP can be avoided by careful placement and creative use of traditional PBXs, VoIP gateways, and other systems. If, for instance, a country does not allow VoIP calls, placement of a VoIP gateway in an adjacent country that does allow VoIP calls will allow calls to be turned into traditional TDM calls before crossing the border into the restrictive country. Regionalization of gateways also allows concentration of support staff and may result in overall higher system availability and lower support costs.

145

Chapter 5 Contact Centers/Call Centers One final area of the utmost importance to many organizations, but not addressed explicitly in the IP Telephony project map, is contact or call centers. Many organizations, be they commercial, academic, or government, very survival rests on their contact centers and the ability to do outbound calling to solicit customers, while many other organizations use contact centers for internal support or to provide aid to customers and prospects. Either way, an organizational shift to VoIP opens the doors to many contact/call center capabilities that are too difficult, too expensive, or not technically feasible using older technology. Special Considerations and Contingency Planning There are many special considerations in shifting call center operations to a VoIP platform. These include all-VoIP or hybrid traditional/VoIP, off-shoring the call center to another country, and on-shoring (which is the outsourcing of the call center to in-country providers and, possibly, distributed call centers that allow operators to take calls from their cubicles or their home). The distribution of operators over such a broad geographic area as is facilitated by call takers working at home also alleviates much of the contingency planning that must occur with a single call center where all call center functions are concentrated in one place. Phased Implementation and Call-Taker Training Part of the migration of call centers to VoIP should involve a safe, secure, phased implementation to the eventual hybrid or all-IP network. Training can often be accomplished online using network-based training and collaboration tools, further increasing the financial and operational benefits of a move to IP Telephony. Planning for migration of a call center to VoIP can best be achieved as a completely separate project from the rest of the migration.

Summary This chapter, though it couldn’t provide every implementation and migration detail applicable to every environment, has hopefully provided a foundation on which you can build a successful implementation. Combine the information provided in this chapter with your own experience and knowledge to synthesize new ideas, new approaches, and new ways to be successful in your IP Telephony project. The next chapter will focus on how to keep things running smoothly after the migration and implementation. Much time will be spent on the “tools of the trade” that allow you to monitor your networks and ensure that they remain within the guidelines established within the SLA.

146

Chapter 6

Chapter 6: Ongoing Operations Most, if not all, successful start-up companies are founded by creative, energetic, entrepreneurial individuals whose 16-hour days, eating at the desk or on the run, decisions on-the-fly and out-ofthe-box thinking that could not possibly be sustained over the long run, are replaced by a new team. After the “start up” phase, top management positions are occupied by less creative, more down-to-earth, and more operations-focused personnel whose job it is to standardize operations and run the company going forward. Companies with foresight often move the founders into key roles of business development or product improvement. Those companies that do not do so often stagnate without the enthusiasm and commitment of the original team. This transition happens as often in Voice over IP (VoIP) projects as it does in entrepreneurial start-ups and for many of the same reasons. The job of creating a project, selling it to management and users, and making it happen—in any size organization—is different to the job of running the system day-to-day. It is the rare individual who can fill both roles. If your job is to take the reins, create harmony from chaos, and run the new system day-to-day, this chapter is for you.

Network Operation The first step as you emerge from the implementation phase and enter the operations phase is to go back to the beginning and review the original plans, goals, and objectives and to ask several questions: •

Are the original goals and objectives still valid?

•

Have technologies changed significantly?

•

Are standards more mature, and does it matter?

•

Have laws or regulations in any of the areas in which you operate changed in any important ways?

•

Have your operating procedures or business environments evolved in a way that impacts your new telephony system?

It may have been several months or longer since the VoIP project was first conceived and planned. Things may have changed: the business direction, the services your users need, and costs, technologies, and products. This review step should also include input from management and users to ensure that all current views are considered.

147

Chapter 6 Once you have validated the original goals, or modified them to reflect any changes, it is time to review the Service Level Agreement (SLA), or agreements, to ensure that they reflect the original goals and objectives and to make adjustments to the SLAs, if needed, in a similar manner to what was done to the project objectives. Once this administrative step is completed, it is time to schedule a review cycle to periodically assess the validity of the project goals and set new goals based upon the network optimization process that will be ongoing in parallel with the operations cycle. The important question is, “How often should operations be reviewed?” Some organizations are fairly static and have, in fact, been doing much the same job in much the same way for many years. Those organizations might very well be able to have a comprehensive review scheduled annually if one is not triggered sooner by the optimization process. Other organizations that are more dynamic and are undergoing more rapid change—or growth, moving into new markets, evolution of processes and procedures, or similar changes—may be at the other end of the review spectrum and might, at least for the first year or so, want to conduct a formal review monthly. The timing is really up to the organization and their needs, but the key is to get it on the calendar and move on to operations management. Using the SLA as a guideline, take a closer look at network capacity, reliability, security, voice quality and call quality, growth management, and how to measure and monitor the network. Capacity, Performance, and the Broadband Flip Anyone who has been in networking before the start of the low-cost bandwidth era is very focused on capacity issues. The industry veteran most likely has a genetic predisposition to focusing on optimizing bandwidth for purposes of capacity and cost savings. The reason for this is very simple: prior to the “broadband era” bandwidth was outrageously expensive and bandwidth demand was calculated and optimized for capacity in much the same way conveyor belts in a factory are sized: getting the most done with the least resources by filling the conveyor system up as full as possible. But, for several reasons, this is no longer the case and you must now do the “broadband flip” For lack of a better label, I will refer to what came before the broadband era as the narrowband era. The narrowband era was characterized by individual circuits, or channels, that were dedicated to the use of a single communication, be it a stream of voice, data, or video bits or packets.

148

Chapter 6

Figure 6.1: Narrowband bandwidth calculation.

In narrowband, a general rule of thumb, the two-thirds rule, or a similar approach is often applied. When utilization grows to about two-thirds of the capacity of the circuit, or channel, it is time to request a larger circuit. If a network manager budgets properly, tracks the rate of growth, and knows the installation lead time for a new or upgraded circuit, the management of this process can be fairly straightforward. For instance, as Figure 6.1 shows, if a remote site is: •

Connected to the central data center by a 9.6Kpbs circuit that was averaging a utilization of 5.2 kbps in January.

•

Utilization was growing at a rate of 10% per month.

•

It takes 3 months to order, install, and test a new 19.2Kbps circuit.

In this case, simple math shows that the new circuit should be ordered no later than April 1st, and probably sooner, to be up and running by August 1st, before demand exceeds the capacity of the 9.6 kbps circuit. Lacking unforeseen growth or upgrades or changes unreported by the user groups, the system worked. Ahhh, life was so much simpler back then…but not necessarily better. To get from “back then” to “right now,” you must do the broadband flip. Stop thinking about bandwidth for capacity and begin thinking about bandwidth for performance. Consider a traditional, channelized, narrowband T1: it has 24 individual channels each capable of carrying exactly 64,000 bits every second. In fact, the type of sharing, or multiplexing, as the engineers prefer to call it, that the narrowband T1 uses—Time Division Multiplexing (TDM)—gives each of up to 24 simultaneous users the impression that they each have a connection that is dedicated to their own use and, in fact, they really do. Figure 6.2 represents the narrowband T1. Let’s take a closer look and ponder some possible scenarios and how they might play out in the channelized narrowband world. In the hypothetical T1 that Figure 6.2 shows, the darkest of the 24 channels, such as the topmost channel, are both configured/reserved and are in use at the current moment. The lighter colored channels, such as the bottom-most channel, are configured/reserved but are not currently being used. The white channels are neither configured, reserved nor in use.

149

Chapter 6 What if the top channel needed more than 64 kbps of capacity? Could it share the channel below if that was already configured/reserved? No, it could not. Could it use any of the idle channels? No, not dynamically. To do so would require a rather lengthy reconfiguration process. Is it possible to use more or less than 64 kbps for a single connection? Yes, but not dynamically: a lengthy reconfiguration and provisioning process is required. The point is, narrowband connections are “nailed up” and not capable of changing dynamically.

Figure 6.2: Channelized Narrowband T1.

And, what about capacity? How is capacity adjusted? Capacity is increased in groups of the basic 64 kbps channels, called DS0s, up to large capacities, as shown in Table 6.1. When the demand exceeds 24 individual channels in a single group, called a T1, the T1 can be replaced with a T2, which can accommodate up to 96 simultaneous 64 kbps connections. It is rare, however, to encounter a T2, as it is more customary to make the jump in most cases directly to a T3, which is capable of supporting as many as 672 simultaneous connections, or a fractional T3 comprising some number of T1s or T2s. The next step before the Synchronous Optical Network (SONET) is a metallic T4, which can accommodate up to 2016 simultaneous connections. The system was built to accommodate progressively larger numbers of the same size, 64 kbps, connections, not to allow for greater bandwidth, as broadband does. SONET, as defined by GR 253-CORE from Telcordia, is a method of communicating digital information using lasers or LEDs over optical fiber

150

Chapter 6

Table 6.1: North American (Narrowband) digital hierarchy.

Figure 6.3 shows the narrowband T1 from Figure 6.2 reconfigured for broadband use. Instead of being carved into 24 identical small channels the non-channelized broadband T1 is a single large channel that can be shared by multiple connections, possibly more than 24, but the key point is that the full single channel is dedicated to a single connection at a moment in time. How is this accomplished? In the narrowband T1, we could associate specific bits with specific channels by their positions. The first group of eight bits read from the wire belongs to the first connection; the second group of eight bits belongs to the second connection, and so on up to 24. Any idle or non configured channels have filler bits to maintain the positioning and this process is repeated 8000 times per second to deliver 64 thousand (8 x 8000) bits per second per channel.

Figure 6.3: Non-channelized Broadband T1.

151

Chapter 6 In the case of the broadband T1 shown in Figure 6.3, bits are not associated with their connections by their position but rather by additional protocol bits that are used to create frames, packets, or cells, which are three different types of electronic containers for transporting broadband bits. Broadband is actually not as efficient as narrowband for transporting multiple identical channels of information, but that is rarely the objective anymore. Broadband is good at dynamically allocating bandwidth resources to the rapidly varying demands of multimedia such as voice, data, and video. In Figure 6.3, it is also possible to observe that idle capacity is not trapped in individual connections that cannot be dynamically shared, as it is in narrowband. In broadband, all capacity can be used dynamically and idle bandwidth only occurs when all information has been transmitted for all connections. And what about capacity? Broadband capacity is enhanced by increasing the size of the “pipe,” not by increasing the number of same-size pipes in some sort of bundle as is the case with narrowband. It is possible to configure narrowband connections—such as T1, T2, T3, and T4—for broadband access, but it is also possible to use Digital Subscriber Line (DSL), cable modem connections, metro and wide area Ethernet, and other broadband services to allow you to get as close as possible to the right size “pipe” for both capacity and performance needs. This is where you must make the big leap from considering bandwidth for capacity—though capacity is still a consideration—to considering bandwidth for real performance and doing the broadband flip. As Figure 6.4 shows, increasing the capacity of a narrowband circuit increases the aggregate capacity but does nothing for transmission performance. Regardless of the number of 64 kbps connections bundled together, the individual channel capacity does not change. Figure 6.5 shows that increasing the size of the actual “pipe” and utilizing it in a broadband fashion have an important impact on performance per connection.

Transmission Time for 64K bits (Narrowband Example) 1 DS0 Channel 1 DS0 Channel in a T1 1 DS0 Channel in a T2 1 DS0 Channel in a T3 1 DS0 Channel in an OC-3 1 DS0 Channel in an OC-12 1 DS0 Channel in an OC-48

= 1 second = 1 second = 1 second = 1 second = 1 second = 1 second = 1 second

Figure 6.4: Narrowband transmission times.

Transmission Time for 64K bits (Broadband Example) Broadband T1 = 1/24 second Broadband T2 = 1/24/4 second Broadband T3 = 1/24/4/7 second OC-3c = 1/24/4/7/3 second OC-12c = 1/24/4/7/3/4 second OC-48c = 1/24/4/7/3/4/4 second Figure 6.5: Broadband transmission times.

152

Chapter 6 Table 6.2 and Table 6.3 show the wide range of possible bandwidth needs, per connection, for a variety of combinations of Voice over Packet and transmission protocols. The difference between Table 6.2 and Table 6.3 is the packetization interval. Table 6.2 shows the impact of creating and sending packets every 10 ms, resulting in more packets but a smoother flow of voice that is less susceptible to packet loss; Table 6.3 uses a packetization interval of 30 ms. With a 30 ms packetization interval, bandwidth is consumed less quickly because more voice samples are placed in each packet, thereby reducing the number of overhead bits per sample; however, the connection is impacted more heavily by packet loss as the loss of a single packet will result in the loss of three times as many voice samples.

Table 6.2: VoIP bandwidth estimate with 10 ms packetization interval. Source: Matthew Michels, Nortel Networks, used with permission.

Table 6.3: VoIP bandwidth estimate with 30 ms packetization interval. Source: Matthew Michels, Nortel Networks, used with permission.

153

Chapter 6 These types of bandwidth utilization models can also be used to determine the impact of voice activity detection (VAD), which eliminate packets that are carrying silence. By eliminating silence packets, it is possible to get additional benefits with a negligible impact on quality assuming the VAD system is properly configured and tuned. Proper configuration involves the treatment of the absence of silence packets at the destination and how silence packets are chosen for non-sending at the source. If in the absence of voice packets, no sound is played into the listeners’ ear, as in the earliest implementations, VAD can be very disruptive to the call and perceived call quality, causing the talker, upon hearing nothing from the listener, to pause and ask “are you still there?” Insertion of comfort noise, also known as Comfort Noise Generation (CNG), is vital. At the source, if packets are not sent that are predominantly silence, chopping or clipping of syllables can occur. Not setting stringent enough criteria for packet non-sending will lessen the value of VAD, which can be as much as 4:1, meaning that an actual call using VAD will require 25% of the bandwidth required by the same call not using VAD. Results of this kind can be used to play “what-if games” and to develop realistic bandwidth estimates for different types of connections and, thereby, be able to establish baselines for the operation of your specific network. It is likely that you will end up, with at the very least, two profiles: inside calls and outside calls. But you may very well have different bandwidth requirements for a much more specific set of user requirements, much along the lines of the user profiles developed in Chapter 4. It might be possible to establish as few as four user profiles that are common to all departments, though it is more likely that there will be far more profiles for any given situation. Let’s look at four basic profiles, just to give an idea of the type of feature sets they might need.

154

Chapter 6

Profile

Possible Job Functions

Telephony Features

Dial Tone In-house (extension) dialing Local dialing Long Distance dialing Basic Voice Mail Call Waiting Caller ID Basic Telephony User PLUS: Secretary, senior secretary, telemarketer, senior help desk 3-way Conference calls worker Skills-Based Call Routing Call Forwarding Intermediate User PLUS: Administrative assistant, group admin, senior telemarketer Enhanced Voice Mail (add FAX storage/retrieval and advanced messaging and retrieval features) Enhanced Call Forwarding Multi-party conference calls Senior sales person, senior manager Advanced User PLUS: Multimedia conferences (web plus audio) Enhanced Call Forwarding with Remote Access Telephony Senior manager, technician, developer, product marketing Call Routing Management and manager, senior sales manager Follow-me Features Voice Messaging Instant Messaging Multimedia Conferencing Collaboration Remote Telephony Sales executive, account manager, senior sales manager, sales VP, Remote Data sales director Remote Video Advanced Call Routing and FollowMe Features Office worker, inside sales person, general staff, supervisor, low level manager, help desk worker

Basic Telephony User

Intermediate Telephony User

Advanced Telephony User

Power Telephony User

Multimedia User

Road Warrior

Table 6.4: Sample user profiles and telephony features.

It is clear that many organizations will have additional requirements, such as various skill levels of Call Takers, Call Taker Supervisors, operators, executives, administrative assistants, and so on, but this basic mapping of user profiles to features will show a way of simplifying both the testing and overall implementation issues. So what should the operations personnel do with all this information? Using these considerations, baselines should be established, growth should be observed, and system capacity tracked. Additionally alternative routes should be established in case of failure, possibly with multiple carriers and other back-up plans put into place, as described in the following section on reliability. There are many modeling, simulation, and tracking tools available to help with this task, as even the smallest networks are too complex to be tracked manually. Tools to help with this function are described at the end of this chapter.

155

Chapter 6 Reliability Establishment of “reliability” baselines and system monitoring to ensure that all components are operating within those parameters is very important to the proper ongoing operation of the network. Reliability must be carefully considered from the standpoint of the impact on the end user. Reliability issues may be considered, and prioritized, in terms of “service impacting” and “non-service impacting” outages. To implement a reasonable approach to reliability, one must consider the percentage of time a system is needed, that the system is actually required; versus the total amount of time the system is available. Beyond the service impacting and non-service impacting categories, reliability must be monitored and addressed in terms of “root cause analysis,” and this must be a key part of training and standard operating procedures. Root cause analysis, in the terminology of the medical profession, allows you to treat the illness, not the symptom. If we stay for a moment with the medical analogy, we would describe the visit of a patient to the doctor. The patient complains about a temperature. Does the doctor tell the patient to lie down in a bath of ice water to lower their temperature or does the doctor ask further questions to ascertain why the patient has a temperature? Remarkably, many call centers and help desks are designed to get the “patient” to fill a bath with cold water. These help desks are usually characterized by setting goals for the call takers such as “closed tickets per hour” or a similar metric. The call centers and help desks that will go to the next step are usually setting goals for their call takers based upon user satisfaction and solving user problems. They will ask the next diagnostic questions and consider the systemic issues of any problem—and that is what must be done. Reliability is related to the “end-to-end view”—the closer a problem is to the end user, the fewer users that problem will impact. The other side of that example is that the closer a problem is to the core of the network, the more users it will impact. For example, a problem with an individual user’s SIP phone will only impact that user’s ability to make and receive calls. Their voicemail will still be working, but they will not be able to engage in interactive conversations. Although this does represent a problem, the scope of the problem and the set of users a problem impacts are limited. However, if a problem affects the VoIP call server, it will impact all users of that server. These considerations should drive the investment of time, money, and effort to prevent problems and your investment in mitigation when problems do occur. Security An important part of ongoing operations is the security of the entire system generally and specifically; in this case, the Voice over Packet services. There are three distinct areas of concern relating to security of voice, specific security threats and vulnerabilities within each area. In hybrid systems, combining traditional systems and Voice over Packet, the number of distinct areas of concern increases substantially. As Figure 6.6 shows, the three areas are the phone itself, the network, and the server. Within the network, there are also three areas of vulnerability: the phone access to the network, the network backbone, and the server access to the network.

156

Chapter 6

Figure 6.6: Areas of vulnerability for IP-based voice systems.

Let’s start a discussion of security from the phone end of the connection. What many people forget is that Voice over Packet-capable phones, regardless of connection type, are actually computers running specific applications. Very often, the phones are programmable in some generally available programming or mark-up language, such as the eXtended Markup Language (XML), and can be reprogrammed as easily as they can be programmed. The types of vulnerabilities, therefore, are very similar to vulnerabilities in the server with the exception that security issues with individual phones may impact one or a small subset of users while server breaches impact all users of that server. Phone and servers are susceptible to three types of attacks: malicious software, or malware-enabled attacks, as they have come to be known; content compromise; and Denial of Service (DoS). An additional consideration, even though most organizations consider VoIP and its related services to be “free,” (organizational telephony services are about as free today as 800 service was free on traditional telephone networks)—is that it’s not. Someone has to pay for the bandwidth and, maybe more importantly, the time of the person using the service. Many organizations are shocked when they do an audit of the protocols running on their network and find a large percentage of their bandwidth is used by personal Skype, unauthorized NetMeeting, and other non-approved system uses. These are often the silent destroyers of response times and performance that chokes off organizational productivity. Malicious software, or malware, is software that is surreptitiously downloaded to a phone or server that does things of which the user is unaware and would probably not approve if they were aware. Malware can be as simple as compromising dialed phone numbers or calling directories by sending them to an unauthorized third party or disrupting normal operation of the phone, such as not ringing or selectively not ringing, or allowing unauthorized use of the phone. Content compromise can allow access to the voice communication by unauthorized third parties and can possibly overcome encryption methods by allowing the unauthorized third parties access to encryption keys. DoS attacks are the simplest type of attack and simply block the user from establishing all calls or specific calls or from using certain system features. DoS attacks are the simplest but the most common. Within the “network” portion of the call, the vulnerabilities are typically of the eavesdropping or DoS varieties. There is greater vulnerability, typically, in the user/phone access than there is in the server access and backbone. The backbone is usually the least vulnerable to ad hoc attacks, especially if the backbone is a Virtual Private Network (VPN) provided by a trusted third party.

157

Chapter 6 Another aspect of security that must be managed operationally is systems that are put into place to enhance security—such as firewalls, stateful inspection engines, and Network Address Translation (NAT)—but that often collide with the Voice over Packet systems and prevent their proper operation. With packet voice services playing an increasingly important role in organizational communications, these are considerations that must be made prior to implementation of the security systems or upgrades and enhancements to the capabilities of the existing systems.

Figure 6.7: Voice over Packet system implemented via a gateway.

A second consideration is packet voice systems that are implemented over traditional telephony systems. They share the basic issues with the systems implemented via IP-enabled phones except that the target for malicious software will be the gateway device that connects the traditional telephone devices to the IP network. This arrangement has not proven to be more or less vulnerable to hacking than the situation in which the gateway functionality is embedded in the telephone device itself. There is yet a third situation, which Figure 6.8 shows that must be considered from a security standpoint: the situation whereby a traditional telco network, either public or private, and an IP voice network are interconnected. This is a very common situation that, the call at some point in its duration will be subjected to the security issues discussed earlier relative to IP-based voice and to all of the security issues associated with traditional telephony. Although traditional telephony security issues do overlap with some IP-related issues, the traditional telephony part of the call has some unique security vulnerabilities as well. The issues that overlap have to do with unauthorized use of services, and the ones that are unique are related to these issues. The cost per packet voice call is usually very low compared with the cost of traditional calls, so, therefore, as mentioned earlier, the largest component of the true cost is usually productivity and time costs, as opposed to per minute charges, referred to in old telephone network parlance as “toll charges”—the same term that gives us the word “toll fraud” for the theft of services with those charges associated.

158

Chapter 6

Figure 6.8: Voice over Packet and traditional telephony hybrid.

The toll fraud issue in a hybrid network is actually a double-edged sword: the IP network can provide new, often unsecured access to traditional telephone networks that have toll charges associated, and the traditional telephony networks can provide unchecked access to the IP networks. In addition to newly minted exploits, the old ones still work and are still an issue: Mailboxes that are set up on corporate systems and resold as part of a service package or used by terrorists or for untraceable or difficult-to-trace drug transactions. Sale of long distance, often international, calls via corporate PBXs; and other similar loopholes and backdoors cost organizations billions of dollars a year. This will continue as long as the IP and traditional networks are interconnected, which in many cases will be for some number of decades. What can an organization do? The best bet is to work with your security team, whether they be an inside team or a part of your service provider, to monitor activity and to thoroughly check out any activity that appears to be abnormal—or an unusually high volume of normal activity. Eccentricities and anomalies that should be monitored are unusual protocols, unusually high or low call volumes, appearance or disappearance of encryption, time of day shifts in usage, unusually high or low volumes for a certain phone number or user, and similar patterns that can mean abuse. Restoring software to IP phones and servers, PBXs, and gateways periodically is also a good idea as is using secure versions of the protocols and systems used to install, maintain, and operate them. Using Secure Shell (SSH), for instance, to maintain gateways, as opposed to the common Telnet protocol that is available to every hacker, is a good idea as is using secure versions of SNMP, OSPF routing protocol, and other fundamental workhorse protocols used to run the networks that carry the increasingly important Voice over Packet traffic. These protocols have many benefits when compared with their insecure variants in that they employ encryption, tunneling, and other security capabilities to ensure greater security of the applications.

159

Chapter 6

Call Quality and Voice Quality We have previously, and vigorously, defined call quality and voice quality and described the difference between the two. Nowhere is this distinction more important than in the operations phase of the Voice over Packet network’s life cycle. Call quality is directly related to QoS and its underlying methods and measurements. Call quality and QoS measure and attempt to optimize, in some cases dynamically, packet loss, delay, delay variation, and system availability. Voice quality, however, is directly related to Quality of [User] Experience (QoE) and is very tightly coupled with the opinion of the humans using the system. QoE is only loosely coupled with QoS even though it is an automated approximation of human QoE opinions derived from the statistics of QoS upon which Service Level Agreements (SLAs) and network-based alarms are based. The reason for this is that QoE scores are based upon the opinion or averaged opinions of one or more humans. Those humans are subject to a variety of factors that are impossible to model and reproduce mathematically, but those humans are also not available to judge voice quality on a given call, so it is important to use automated systems. The benefit of automated systems is that, although they do not exactly reproduce the human scores, they are consistent and always available. The drawback is that they only roughly approximate the humans’ opinion of the voice quality. The two types of testing are non-intrusive and intrusive. Non-intrusive testing does not require a special test call and is based on actual conversations or is calculated from metrics gathered during an actual call. Intrusive requires a special test call and is based on comparison of source signal with the signal after transmission. Both are valuable operationally but the non-intrusive testing will raise a realistic red flag sooner as it relates directly to the quality of the voice as experienced by the user because it is a set of measurements made on real calls. Intrusive testing not only is artificial in that it is not based on actual calls but also can place an additional burden on a network for the test calls themselves that may distort the test results and even have a negative impact on real-live calls in progress. This is especially true if intrusive testing is performed at the most desirable time to do testing: when the live call load on the network is particularly high and nearing upper limits. Non-intrusive tests include the Mean Opinion Score (MOS), derived MOS, the E-Model (ITU-T G.107) R value, and, more recently, ITU-T Calculated Planning Impairment Factor (ICPIF) Scores (ITU-T G.113). MOS testing has been adopted from traditional telephony, and historically has been, at least for North American telco operations, the actual opinion of voice quality from a panel of human judges from central Illinois. Even though one might question whether it is statistically significant, the traditional panel has been composed of 16 people: 8 men and 8 women. The panel of judges is sequestered in a sound-controlled laboratory environment and judges the quality of voice on a variety of systems on a scale from 1 (lowest) to 5 (highest). MOS is useful for human user QoE analysis but is virtually useless for SLAs and network monitoring due to the difficulty in reproducing the results reliably and under a wide variety of circumstances.

160

Chapter 6 The E-Model (ITU-T G.107) is a much newer innovation and is intended as a design tool that predicts average voice call quality using a mathematical model that takes into account the estimated impact of delay, delay variation (also known as jitter), packet loss, and performance of the codec (the voice coder that translates human speech into bits at the source and decodes the bits back into analog waves at the destination). The result of the E-Model calculation is the R factor or R value. The R value is an estimated voice quality rating with a range from 0 to 100. 0 indicates lowest call quality and 100 indicates a perfect quality score. From the E-Model’s output, the R value and derived MOS can be calculated, as Figure 6.9 shows.

Figure 6.9: Correlation of E-Model R values to MOS scores for derived MOS.

In actual operations environments, the author recommends a target of 100, which matches to a MOS of 5.0—a perfect and unachievable score. But, like archers, Voice over Packet operations’ personnel need to aim above the mark to actually hit the mark and, in this case, aiming for a perfect score will probably put the voice quality in your network consistently in the R value range of 94 or so. This matches a derived MOS of 4.4, which is in the same 4.3 to 4.5 range generally achieved by most traditional voice networks based upon Pulse Code Modulation (PCM) and TDM transport. The real value of the E-Model lies beyond the design work for which it was originally created and is realized more in the operations area, which is the subject of this chapter. Operationally, EModel scores are based upon measurable parameters found in the network and which can, if needed, be reproduced in the lab. In actual operation, E-Model evaluates Real-Time Protocol (RTP) Streams based upon source and destination addresses, port numbers, and sequence numbers and creates a jitter profile and predicts the impact of the Internet Protocol (IP) bearer on MOS with an 80 to 90 percent correlation to actual human scores.

161

Chapter 6 An additional non-intrusive evaluation technique is the ITU-T ICPIF score (ITU-T G.113). Some manufacturers are beginning to adopt ICPIF for voice quality calculations due to the additional levels of sophistication being required in assessment of voice quality. In fact, from the perspective of VoIP service providers, this author has always maintained that when all prices are equal—and as low as they can realistically go without bankrupting the service provider—other differentiation factors will emerge that will be the basis for purchase decisions and that the human interpretation of voice quality will be one of those factors. We are already beginning to see this in services offered to the public. From an organizational point of view, this has never ceased to be true for two important reasons. The first reason is that in the organizational/enterprise/corporate/agency environment, cost savings are not seen and felt directly by users in most cases: they only see an improvement, degradation, or no change in the quality of voice calls and the related telephony and multimedia services available to them. The second factor is that although most people’s opinion of what call quality is possible, and reasonable, has been strongly, and downwardly, influenced by the use of cell phones, many in the business environment still have a traditional, reliable, high-quality telephone with which to compare new service. Therefore, a lot of work is being done in the area of voice quality, much of it in anticipation of the shift of importance to voice quality and the ICPIF score, available since February of 1996, is a good example of this. ICPIF takes into account the factors that the E-Model does and additionally gives weight in the evaluation to signal attenuation distortion and loss, circuit noise, codec distortion, group delay distortion, one-way transmission time, talker echo, and some additional parameters of special importance. Figure 6.10 shows the rough correlation of ICPIF scores to MOS. It is, therefore, possible to derive MOS values from ICPIF scores just as it is possible to do so using R values.

Figure 6.10: Correlation of ICPIF to MOS.

162

Chapter 6 Intrusive evaluations, while not as valuable operationally as non-intrusive tests, still have operational value. Intrusive tests are often used to perform automated testing during busy hours or non-busy hours, and several companies have systems that are automated and operate free of human intervention. The family of intrusive tests includes Perceptual Analysis Measurement System (PAMS), Perceptual Speech Quality Measure (PSQM), Perceptual Evaluation of Speech Quality (PESQ), and PESQ+. All of these evaluation approaches are derived from PAMS, which was developed by British Telecom as a replacement for human MOS values. Although PAMS and its close relatives that follow do not replace MOS, only approximating it within +/- 10 to 20 percent (a window of up to 40%), they are still excellent for benchmarks and comparisons. This group of tests work by comparing the original analog voice wave with reproduced/transmitted speech using a complex weighting method intended to take into account characteristics important to the human ear. The scale is from 0 to 6.5 with 0 being “perfect” (that is, no difference between waves). PSQM, ITU-T Recommendation P.861, and PESQ, ITU-T Recommendation P.862 and PESQ+ operate very similarly to PAMs but come close to approximating MOS values. Measurement and Monitoring So what is the value of this in operation? The SLAs that the operations group inherited from the design and procurement process should be based, to some degree or other, on MOS. Because it is not practical to line up humans to listen to all calls, an automated method is needed and this, or some other variants—some proprietary and some standard—can be established as the networkwide standard. Non-intrusive tests can be used as the basis for evaluating human user complaints and intrusive testing can be used to anticipate issues and take a proactive approach to fixing them before they reach a level sufficiently bad to generate user complaints and the resulting trouble tickets. Baselines need to be established early in the operational life cycle, consistent with the SLA, and any variation above or below the baseline by more than your tolerance—+/- 5% gives a 10% window that is reasonable in most circumstances—is reported, a root cause analysis is done, and whatever steps are needed are performed to get the observed scores back into line with the baseline values for the network. NetFlow/IPFix

An important consideration for measurement and monitoring is the exact methodology that will be user to implement it. In many cases, organizations opt for bulk statistics or exception reporting through SNMP or Remote MONitoring (RMON), but a methodology that is being standardized and receiving a lot of attention recently is NetFlow, also known, in the standards area, as IPFix. NetFlow is an open but proprietary network protocol developed by Cisco Systems to run on Cisco IOS-enabled equipment for collecting IP traffic information. Cisco routers that have the NetFlow feature enabled generate NetFlow records; these are exported from the router in User Datagram Protocol (UDP) or Stream Control Transmission Protocol (SCTP) packets and collected using a NetFlow collector. Juniper Networks provides a similar feature for its routers called cflowd, which is basically NetFlow 5. Huawei Technology routers also support the same technology, but call it NetStream.

163

Chapter 6 Historically, network flows have been defined in many ways. In the case of NetFlow, Cisco uses the common 5-tuple definition, where a flow is defined as a unidirectional sequence of packets sharing all the following five values: •

Source IP address

•

Destination IP address

•

Source TCP port

•

Destination TCP port

•

IP

The router will output a flow record when it determines that the flow is finished. It does so by flow aging: when the router sees new traffic for an existing flow, it resets the aging counter. Also, TCP session termination in a TCP flow causes the router to expire the flow. Routers can also be configured to output a flow record at a fixed interval even if the flow is still ongoing. In Flexible NetFlow (FNF), an administrator could actually define flow properties on the router. Beyond the Cisco realm, the Internet Engineering Task Force (IETF) is standardizing NetFlow under the title IPFix—IP Flow Information Export—and it is being made available on a wide variety of network products. The gathering and analysis of NetFlow/IPFix data is often best performed on separate management platforms using third-party software of which a wide range is available. Growth Management Growth of a Voice over Packet network is a part of the bigger project of managing growth of a multimedia IP network. As such, it has much in common with the bigger project but also has issues of its own. Growth must be managed in terms of fixed assets and equipment budgets, access bandwidth, usage of the shared bandwidth, and shared resources such as switches, routers, telephony servers and gateways, and numbers and licenses. Fixed Assets and Equipment Budgets

The first job will be keeping track of physical assets, which have both hardware and software elements. The Voice over Packet network will include hardware and possibly software telephony clients for the end users. In many cases, this will include the reallocation of traditional telephony devices such as desk phones to the Voice over Packet inventory. Fixed assets will also include telephony-only systems, such as SIP servers and gateways, and might include an allocation for the part of shared systems, such as routers and switches that are used by voice traffic. Although this is often a secondary consideration on the list of many seemingly more critical items for operations staff, this can be a time-consuming and possibly career-threatening item if ignored. Proper tracking and forecasting of growth includes awareness of pending management decisions regarding mergers and acquisitions, layoffs, new product lines, or anything that can substantially impact the number, and profile, of the user community. The recordkeeping associated with this task will be addressed further in subsequent sections.

164

Chapter 6

Network Bandwidth: Access and Backbone

The second job will be managing the bandwidth pool. Bandwidth utilization issues have been addressed in a prior section and at this point in the life cycle of the Voice over Packet network, calculated values should have been compared with actual values and you should have a clear picture of the bandwidth that is used by each type, or profile, of user. You should also be aware of the number of each type of users and the likelihood of that sub-population to substantially grow, shrink, or be reallocated to a different profile. Recall that an element of the user profile was the degree of mobility, which can impact user access as well as the amount of shared bandwidth they use in the network backbone. All of these factors need to be managed; human resources and management hiring and acquisition of other organizations needs to be factored in and a budget developed. One of the biggest factors that can influence a finely tuned and well-performing network is a shift in users or among and between user profiles. How can this be managed operationally? Most systems, especially VPN systems and others that require user sign-ons, allow resources to be tracked to some level by individual user and, at the very least, by user group. The key is to anticipate the need and to establish appropriate profiles in advance and to use the profiles to establish user groupings. Another benefit of such profiles is that, very often, user assignment to specific profiles can be done using a standard template that substantially reduces the chances of errors and makes adding new users much faster and easier. Shared Resources: Switches, Routers, Call Servers, IP PBXs, and Gateways

Telephony users are typically multimedia users, not telephony-only, though in either case, it is possible to determine the amount of shared resources that a user consumes. This is a very important growth planning and management element and should be tracked, reported, and reviewed regularly. Specific alarm thresholds being set to send up some sort of electronic signal flare should utilization approach or exceed thresholds between reviews. Shared resources to consider include the switches and routers that are responsible for transmitting and receiving all traffic and the call servers, IP-PBXs, and gateways responsible for handling voice traffic. Numbers and Licenses

Resources that are often not considered, but must be, are numbers and licenses. The two main numbers that are required are phone numbers and IP addresses. The licenses that need to be managed are any software licenses that are charged per user, per seat, per port, or on any other incremental basis. Phone numbers, most often, must be purchased from an outside telephony organization so that they may be registered in the proper places and so that appropriate routing may be established in the Public Switched Telephone Network (PSTN). Organizations are strongly advised not to use telephone numbers of their own creation, even inside what seems to be a private network, due to the fact that almost invariably there will be a need to connect to the PSTN and number duplication is inevitable. It is worth the extra cost. This is a bit less true with IP addresses because NAT can be used to map internal, potentially conflicting, IP addresses to external, registered IP addresses, but this point should be very carefully considered. It is possible to create and manage an inventory of available phone numbers and IP addresses, to assign them as needed, and return them to the inventory for reassignment when they are not in use.

165

Chapter 6 Licenses, likewise, should be inventoried and returned to the inventory for reassignment when not in use. When a user leaves or is terminated, the company almost always remembers to get that user’s keys and delete their passwords. A part of this procedure should also be to return any licenses to the common inventory. Licenses cost money and are needed to operate most Voice over Packet systems. Recordkeeping and Documentation This is the boring but inevitable part of any operations environment: Keeping track of the assets. Although boring, and inevitable, this is still a very important function and one that must be kept up to date as information may be requested at any time. Fixed Asset Accounting Fixed assets include all PCs, laptops, palm devices, routers, switches, Power over Ethernet racks, and just about anything that one can touch. These are the assets of the organization that have been entrusted to the care of the network or IT department for the benefit of the company. These assets must be managed. In many cases, the finance department will insist that you use their fixed asset system or, at the very least, provide your data in a manner that can be input to theirs electronically. The problem with using a generic system is that it often does not capture specific characteristics that are desirable to have to manage your network properly. Information such as release levels, software and firmware revisions, and other key characteristics are often lost. It is also true that many of the automated tracking and inventory systems for networked components are not directly compatible with generic organizational fixed asset systems and must be put through an intermediate step, such as a Comma Separated Value (CSV) or Tab Separated Value (TSV) files, Excel spreadsheet, and/or XML output. If you have not inherited a fixed asset system nor have one already in place, you must procure one as part of the project. The considerations for a fixed asset accounting system closely mirror the considerations for any new software application. What are your basic system requirements? Will the system run on servers with existing applications or on a different platform? What hardware does your MIS or IT department recommend? What operating systems can be supported? How many users will access the system and what are the security requirements? Will shared access over the network be allowed? What additional user rights and permissions must be configured and enabled to support this? Will the new fixed asset accounting system be a separate, standalone system or an add-on module to your current accounting system? If you require a standalone system, it is important to understand the system’s interface capabilities. Will it interface with general ledger accounting modules and tax applications to avoid duplicate entry of data? Does the system allow data import and export? If so, what import and export formats does it support? If you require an add-on module to your current accounting system, you must make sure that the add-on module will interface with your current accounting system.

166

Chapter 6 What do you want in terms of reporting capabilities? Standard reports should include depreciation, such as projected depreciation and depreciation method comparisons, to aid planning. What kind of tracking does it do for financial reporting? Also be sure to investigate the ad hoc reporting capabilities. Is there a limit to the number of user-definable reports you can create? Is it difficult to create ad hoc reports? What are your data entry and depreciation needs? Many packages offer features that significantly reduce fixed asset entry and calculation complexities. Is the user interface easy to use? Does it offer standard templates to ease data entry? Does it offer automatic data validation? Does it offer automatic calculations on a periodic and daily basis? Does it allow you to customize the information you want to enter? How many depreciation methods are supported? What are your fixed asset tracking and management requirements? Whether you are a large or small company, it is important to understand your asset tracking and management needs. Does it keep physical track of where assets are located? Does it track multiple locations? Does it manage multiple companies? Does it offer bar code capabilities? How does it track assets of mobile users such as cell phones, PDAs, notebook computers, and similar devices that may be on the road as much as their human users? All of these concerns, and more, must be addressed in close collaboration with the accounting department. Shipping, Warehousing, and Tracking Shipping, warehousing, and tracking of assets are critical. In many cases, devices and systems are ordered and shipped directly to the end user, very often being charged to the user’s personal or company credit card and charged back to the company through an expense reporting and reimbursement procedure. Are these assets tracked? What is the dollar value threshold for your organization that must be tracked? How much money is effectively lost each year by not doing proper tracking? Or maybe assets such as SIP phones or VoIP over WiFi devices are centrally purchased and shipped to the user from a warehouse. How are these assets tagged and tracked? Repair/Refurbishment/Return to Service/End-of-Life Another area of consideration is repair, refurbishment, and return to service. How is this tracked? Are users provided with a loan or replacement system? How often are systems refurbished, either via software refresh and upgrade or cosmetically? What happens to assets at the end of their accounting life? Are they sold, given to schools or charities, or recycled?

167

Chapter 6

Administration In most organizations, operations consist of some level of operations management, often, in larger organizations, headed by a Vice President of Operations or similar function. Below that rectangle on the organizational chart, the line usually splits into at least two directions. One direction is the operations group of engineers, technicians, and often Help desk people who manage the technical operations and keep the hardware, software, and network running. The other line goes to a group of administrative people who do the paperwork, manage the budgets and inventories, and, basically, keep the group running that keeps the services running. There are many different functions of that second group, but we will consider some of the primary functions here that may be somewhat different or have some unique characteristics for Voice over Packet networks. End-User Cost Allocation I well remember the first time I heard the term “funny money.” Besides being a two-word poem it caused me to ponder what was so funny about it. Upon further inquiry I learned that computer usage—at the time access to a mainframe computer over 9.6 kbps leased lines—was used to allocate costs to departments but that paper checks were never written nor did real money change hands. The transactions were purely bookkeeping transactions but, to the department heads, were real nonetheless. If they had a surplus of money in their budget, they might not get the full allocation next year; if they ran out of money, they might not be able to access the computer system, and so on—normal budgeting issues. Many organizations still track usage of telephony systems, both the variety of usage that results in monthly charges and usage that is part of a fixed cost system such as a private IP network or flat-rate VPN. The most logical, and common, metric to track is MOU or Minutes of Use. This is the easiest to track in most cases but might not provide a clear picture of the assets, or parts of shared assets being consumed, such as bandwidth. Two other facets of the MOU tracking can be the quality, or Class of Service (CoS), of the call and the bandwidth being consumed as these other factors should have a “cost” associated with them. An additional reason to use MOU as the basic unit of measure, at least while the telephony network is still a circuit/packet hybrid using both the VoIP and PSTN networks, is that MOU will provide a common tracking and cost allocation metric that is used across both telephony platforms. Detailed Usage and Billing There is an inclination when making the move to IPT, and VoIP specifically, to abandon many prior financial controls and recordkeeping. Detailed usage and billing, however, is still required in many organizations for needs that go far beyond “funny money” accounting needs. In the case of professional services, detailed usage information supports client billing: not for the phone call but rather for the professional services delivered over that phone call. Detailed usage is also used to track sales progress or the efficiency of phone support, not to mention use as a management tool to review how an employee spends their time and to question non-business use—an item that is ever smaller in terms of network services costs but ever increasing in the cost of the human time wasted.

168

Chapter 6 Vendor/MSP/Carrier Billing An important area of any operation is ensuring the accuracy of bills, and nowhere is this more important than in the area of recurring charges—those charges that come up each and every month and must be paid over and over again. A problem with the bill rarely gets fixed and just becomes a part of a bigger problem over time. It is critical, for budgetary purposes, to review all vendor, managed service provider, and carrier bills each and every month against the orders for the services that appear on those invoices. The same holds true for periodic equipment invoices, though one-time invoices for purchases are more likely to be audited at the point of receipt of the product. In many instances, organizations use the move to a Voice over Packet system as an opportunity to clean up their billing that has slowly become inaccurate over time. They consider the new packet voice system as a clean piece of paper and meticulously account for each and every new charge and track it. When employed by a new operations manager who is looking for a way to sort out the mess they have inherited from a predecessor, this approach has a reasonable chance of succeeding, assuming the new operations manager has the discipline to keep the new system updated. When used by an existing manager who created the mess in the first place, this is often a formula for assured failure: if this manager could not manage the old system what changes are being made to ensure that they can manage the new system? Cost Reduction and Network Utilization Maximization The job of looking for ways to reduce costs and maximize network utilization is the task of network operations. The job of putting those ways into production and modifying the underlying operations procedures is the job of network optimization and is covered in the next chapter.

Provisioning Traditional telephony operates on a system of OAP&M—Operations, Administration, Provisioning, and Maintenance. In effect, these procedural steps are outlined in this operations chapter. The “P” part of the task—provisioning—includes setting up, configuring, installing, testing, and “turning up” any new service or feature a user may need. This includes the initial installation; any subsequent moves, adds, and changes; and the refresh of the technology and audit of the installed system. Initial installation includes proper configuration, verification, and testing and should also include a security assessment to ensure that installation of the system is proper from a security standpoint and does not introduce any security vulnerabilities into the system. Initial installation should also include user training and execution of any asset tracking documents or other financial recordkeeping.

169

Chapter 6

Maintenance Once the system and its components have been placed into service, the maintenance cycle begins. Maintenance of hardware, software, and QoS and user QoE levels is a key function of ongoing operations. Basic maintenance functions include managing moves, adds, and changes and handling technology refresh and periodic audits. Moves, Adds, and Changes Moves, adds, and changes (MAC), including de-installation and de-commissioning of equipment, are the things that absorb a large part—in some cases, the majority—of the operations budget of any organization. VoIP systems, using Dynamic Host Configuration Protocol (DHCP) and other dynamic configuration options, were supposed to solve the MAC problem and allow users to easily relocate to any location where Internet access was available— inside or outside the company—and to simply connect. Although users of public VoIP services have, in fact, achieved this level of dynamic connection, many within the enterprise and agency networks have failed to do so. Inside the walls of the organization, many times, are internal defensive mechanisms—firewalls and similar systems—deployed in a defense-in-depth arrangement that often prevents easy, user-initiated MAC. An additional tool, but one that also adds complexity, is a VPN, such as those based on wide area Ethernet services and MPLS, and Virtual LANs (VLANs) to carry multimedia traffic. All of these areas must be considered in the light of your specific implementation and security needs. Technology Refresh Periodically, a technology refresh will be required. This may be as simple as ensuring that software and firmware are up to the most current release level or may require hardware upgrades or replacement as well. The rate of technology refresh should not be based on the rate of change of the technology or availability of new features but, rather, should be based on the need for the new technologies or features. For example, if it has been determined that a specific profile of VoIP user can get all the functionality that they need from plugging their old desk phones into Analog Terminal Adapters (ATAs) but new, low-cost SIP phones become available, the consideration to replace the ATAs should be relative to the basic operations needed by that profile and not based upon the additional features, however desirable they appear to be. A refresh is just that—refreshing capabilities that exist and have been previously approved not an upgrade or adding additional functionality. Audits Periodic audits must occur to verify both functions and assets in place. Audits should include functionality, security, hardware, software, services, and circuits. A comprehensive periodic audit, aided by automated auditing tools, should be performed independently of the requirements of the finance department. Audits of Voice over Packet systems should be sure to include all components and end systems, including those possessed by mobile users and users in their small office/home office environment.

170

Chapter 6

Operations Tools Decisions must be made about the tools needed, the amount of graphical reporting required and the testing methodology. Whether this is active and non-intrusive during the call, or intrusive and separate from live traffic as well as tracking relative to the SLA and/or other baselines, and the amount of integration that is desirable. Integration Generally speaking, the complete operations picture will require several tools—both on the technical operations and financial/budgeting sides. Some degree of integration should be expected between the different tools and may require some custom development work in order to get a full, comprehensive solution. This should be anticipated and worked into any plan that is developed. Operation Ongoing operation of network monitoring and management tools will require proper budgeting for training and planning to ensure that qualified personnel are available to use the systems and will most likely be integrated with ongoing network operations. The special training and skills needed to get the most out of the system will be the biggest concern. Besides considering employees, this is also a task that can be outsourced to a third party and managed by the organization.

Summary Chances are that if you are responsible for the ongoing operation of the Voice over Packet network and its associated services, you have inherited something that is a bit chaotic and not quite fully formed. That is the nature of new things. If the job has been done correctly, many of the important operational considerations—baselines, SLAs, accounting, and asset tracking systems—will have been made and some initial steps taken to equip you to do your job. If not, the scope of your job is bigger. The job of the operations group is less glamorous in many ways than the job of designing and procuring a new system, wrestling it into place, and making it work—but is no less important. It is the job of the operations group to ensure that the system performs to specifications consistently—to keep humming along and doing its job day and night, weekdays, weekends, and holidays without glitches or errors, through upgrades and surges in user demand. The next chapter will consider the task of network optimization: improving the network operation in some way and providing new baselines and operational guidelines for the operations group to maintain in, hopefully, a never-ending loop of improvement—operations, improvement, operations…on and on and on.

171

Chapter 7

Chapter 7: Optimization This chapter discusses the process of reviewing the performance of your IPT system and setting new standards and benchmarks. In other words, taking what is already a world-class system and making it even better. This chapter will make the distinction between operations, the objective of which is to keep things as they are, and optimization, the objective of which is to modify the telephony solution in a manner as to improve its operational characteristics. The chapter will also make a distinction between optimization and troubleshooting and repair, the objective of which is to make something that is not operating or not operating within guidelines to work once again according to the guidelines. In fact, the underlying assumption of troubleshooting and repair is that some capability, system, or feature operated properly at one time and that something has caused it to no longer work as desired. This differs from optimization in that operation in an optimized state is desirable but the system, feature, or capability never operated within that state before but will after the optimization. In fact, the optimized condition will become the new baseline for proper operation and the next review cycle may improve further on prior work in a never-ending loop of improvement.

SLAs and the Scope of Optimization Because the SLA represents the classes of services that will be provided by the network and because it was, or at least should have been, the focal point of formalization of the needs for the network and IPT services, the discussion will return once again to the SLA. Many of the most important optimization topics can be addressed via the SLA, and those that cannot, become candidates to upgrade the SLA to meet current and anticipated needs.

Figure 7.1: Scope of optimization and the SLA.

When SLAs were originally introduced, they were a tool of traditional service providers that offered services such as Frame Relay, ATM, and Internet access. The “service” for which the level was agreed to was a network-based service. Therefore, the points between which the SLA was defined were within the range of control of the service provider. It was only the rare case in the early days of SLAs when an SLA included the access circuits: SLAs were almost universally measured within “the cloud,” as Figure 7.1 shows.

172

Chapter 7 The view of the SLA covering just the Layer-2 aspects of network services has undergone quite a bit of change since the early days. SLAs today are, increasingly, service-aware IP SLAs or even Multiprotocol Label Switching (MPLS) SLAs that add value to the IP infrastructure. What is required for a comprehensive and effective approach to optimization is to extend the scope of the SLA. The new definition, the domain of optimization, should not include only the Layer-2 services—which, when strung together end to end, create a path over which applications bits flow—but also the emerging concept of the application-aware IP SLA and a new, user-centered view of the end-to-end service. In this manner, the concept of the SLA is being maintained, while the scope is expanding to meet the real, service-oriented needs of optimization so that you can maintain alignment of service-level objectives and the SLA. There is an entire industry dedicated to application monitoring and optimization. This industry is focused on the use of the actual application such as SAP. There is also an entire industry dedicated to monitoring and optimizing the packets, frames and, ultimately, bits, that flow over circuits. What is being proposed is something somewhere between the optimization of network services, in our case IPT, including the operation of all signaling and other associated functions that support the connection, transmission, and disconnection of packet voice calls.

The Optimization Cycle By this point, you have set up a world-class packet telephony network, you have assured that, at the very least, the telephony service you are providing meets users’ quality expectations and delivers the same basic services as the traditional telephony network it is replacing. It is also quite possible that you have added a number of important features and applications, such as unified communications, to the new network. The new applications make the users more effective in their jobs or make special new capabilities available, which is the real reason to go to all the trouble of implementing a new network. After all, they already had dial tone. By this point, you have also done all the broad, sweeping, system-wide things that can be done to ensure proper operation of your packet telephony services. It is, therefore, time to take a closer look at a number of very specific areas and to optimize what you have created. Optimization is an ongoing process that is required to keep network costs in check and to avoid the demands of the users running ahead of the capabilities and budgets of the network. IPT networks have a tendency to consume huge amounts of resources and must be continually managed. Once the network is in place and IP-tone has replaced dial tone, the next step might be static image cameras on phones and then full motion, ad hoc, video conferencing, also referred to as video-tone. Who drives the changes? How much control does the internal or external service provider have? To what extent is the SLA considered? Is the SLA changed to reflect new needs or are the new, emerging requirements somehow shoe-horned into current categories? Independent of the answers to these questions, the network and its services must constantly be optimized, and, beyond the administrative questions raised here, the optimization process will be the subject of this chapter.

173

Chapter 7

Figure 7.2: The IPT optimization cycle.

The Use of Software Tools The software tools that will be employed for optimization include both the measurement and monitoring tools used in day-to-day operations as well as specialized modeling and simulation tools last employed at the time of the initial design process. The use of software tools is emphasized within the optimization phase due to the ubiquity of the tools within the network— the fact that they are constantly testing and monitoring many aspects of system operation, the repeatability of the tests and the ability to play “what if” games and simulate different scenarios in order to understand optimization options prior to putting them into operation. Tool Ubiquity Software tools can monitor a complex set of metrics across a network simultaneously. The implementation of monitoring tools must, of course, have been accomplished during the implementation phase as retro-fitting an active network with monitoring tools is far more complex, time consuming, and expensive than doing so up front. However, if the proper tools are not in place, it is time to circle back and put them in place. Both intrusive, often called active, and non-intrusive, often called passive, testing will be employed and will provide baseline operational data on the aspects of IPT system operation that will be employed. It is also important to have underlying network infrastructure statistics; however, the two families of tools that you need should be clearly understood and their roles defined. On the one hand, specialized tools are required that monitor specific characteristics of real-time services; in this case, IPT. For IPT, the types of statistics that must be monitored include calculated MOS, delay, delay variation, E-Model R Value, and other metrics of importance specifically to the real-time voice class of service. It is also necessary for these statistics to be made available in a high-level summary with graphics—in effect, a simple network-wide report card—as well as at the individual call level so that the impact of optimization decisions on individual calls can be assessed. On the other hand, you still need to understand the operational characteristics of the underlying delivery system: the network. The second set of statistics you are seeking comes from traditional network monitoring and management tools and includes availability of network components, trunk congestion, router buffer and queue management information, and similar types of metrics.

174

Chapter 7

Constant Testing and Monitoring Constant testing and monitoring is as important to ongoing operations as it is to optimization. In operations, it is desirable to see how the network, and overlaying services such as IPT, is performing in an attempt to ensure that they are performing within established guidelines. Optimization seeks, in contrast, to improve the operational characteristics and, therefore, to reset the baselines to higher standards. One of the aspects that constant testing and monitoring will show is any trends in network operating characteristics that might highlight important changes that must be considered within the context of the optimization process.

Exception Reporting and Filtering A multimedia network of any size will generate far more information than can reasonably be assimilated by administrators working without automated tools. For this reason, exception reporting is used within the operations process to automate the function of identifying when certain aspects of operations are above or below certain pre-set thresholds. If, for instance, packet loss for VoIP connections exceeds a preset threshold of 3.5%, the operations personnel are alerted of an exception condition. This can be done through a variety of mechanisms, such as SNMP traps that cause red dots to appear on the screens of network operators to emails to beepers or IM messages to key operational personnel—and all this is done in real time. Because optimization is not done in real time and, in fact, uses historical data, there is not “exception reporting,” per se, in optimization. The optimization equivalent of exception reporting is filtering. Software tools allow optimization engineers to sift through mountains of data using specific filters to identify exceptions and trends and any specific areas in which anomalies might exist that can be exploited by optimization. “What If” Scenarios The natural outgrowth of being able to work with mountains of real data from a real, working network is the ability to perform “what if” scenarios that will have outcomes that much more closely match the real operating environment. Having real operational data allows far more accurate prediction than was possible during the design phase of the project and, in fact, almost certainly means that the first optimization phase or two will yield much better results than subsequent optimizations. During the design phase, inputs to the “what if” process were either simulated traffic—because no similar real network previously existed from which to capture traffic—or was based upon an IP network that existed before the current network but supported data only and has been substantially changed in the process of implementing the new IPT solution. “What if” scenarios can actually be executed using one or both of two approaches. Ideally, both approaches will be used. The first approach for the “what if” exercise is to have a specific list of characteristics, mostly encompassing the SLA metrics and possibly a few others, and to be looking specifically for ways to improve those metrics. This is the most common approach. The second approach is to rummage around in the historical network performance data looking for optimization opportunities. This second direction will often yield surprising and good results. In this second approach, it is also important to keep an open mind to anything you might find and, in fact, to discipline yourself and your team not to try to guess at or pre-determine the outcome.

175

Chapter 7

Trending and Capacity Planning Optimization includes trending and capacity planning and extends to minimizing infrastructure costs, such as decommissioning excess gateways, lines, and similar un-needed items; maximizing call success rates, security, business continuity; and future proofing the network. Some of the areas to which optimization may most productively be applied are: •

Bandwidth

•

Ports and lines

•

Phone numbers and numbering plan

•

Grooming

•

VLAN capacity

•

VPN capacity

•

Infrastructure

Let’s take a closer look at each category and their interdependencies. Bandwidth Bandwidth is as inexpensive as it has ever been and the price keeps going down; it is worthwhile to invest in bandwidth to ensure a higher voice quality. But, even considering these two points, there is no reason to waste bandwidth. The optimization phase is the time to reconsider several options relative to bandwidth. The first item to reconsider is the choice of coding scheme. Pulse Code Modulation (PCM) has been strongly recommended throughout this guide as the best choice for a variety of reasons, but PCM also requires the most bandwidth of any of the voice coding choices. The optimization process should begin by reviewing the reasons for choosing the specific voice coding scheme currently being used and determining whether it is still the proper choice. In this example, let’s assume that this is the first optimization and that PCM was chosen. What were the reasons for choosing PCM coding for voice? Are those reasons still valid? The first reason is that PCM delivers the best voice quality under a variety of impairments and network conditions. PCM translates sound, not just voice, into ones and zeros for transmission across a digital circuit or packet network and will deliver a voice and “sound” quality that is as near as possible to the circuit system being replaced. PCM-based systems will often deliver a voice quality in the 4.2 to 4.4 range on a 5.0 MOS scale, within the target range for voice quality over a packet network.

176

Chapter 7 Once the transition is made from circuit PCM to packet PCM, it should be easier to wean users over to other, less traditional-sounding codecs. This process can be accomplished slowly over time; it would be very difficult to do all at once. It is also true that newer digital codecs, particularly broadband codecs such as Global IP Sounds (GIPS), designed specifically for use in packet networks are improving constantly both in terms of better MOS and lower bandwidth utilization. After considering the effect of codecs on the bandwidth of individual connections, especially on access or LAN connections, the cumulative impact on shared connections, such as shared network access and backbones, must be considered. A small change on individual user connections, such as their Virtual LAN (VLAN) connection, may seem insignificant in the bigger picture but the cumulative impact can be profound when hundreds or thousands of simultaneous connections are considered. Also remember that the major goal of broadband systems, after a large enough pipe has been provided, is enhanced performance and that the instantaneous performance of a broadband system is enhanced by more smaller packets than by fewer larger packets. In addition, although this is at odds with conventional LAN thinking—a frame of reference built on large quantities of cheap bandwidth going short distances—it is a guiding principle for this IPT deployment, especially if the communicating endpoints are somewhat distant. Ports/Lines Ports and lines are two hardware units of measure—not to be confused with logical Layer-4 ports—that must be optimized. They are also two terms that have different meanings based on context; the two possible contexts being traditional telephony and IPT. In the traditional telephony context, a port is a physical connection into which one would plug a single telephone device or a device capable of connecting single telephone devices in groups of 1 to 24 per physical port. As the use of IPT grows, the use of traditional telephony ports should decline, though they may not completely go away. It is possible, for instance, that a traditional phone service may be maintained for faxing and/or for use with 9-1-1 calls. One such situation exists at Pauling County School District in Georgia and is detailed in the following sidebar.

177

Chapter 7

VoIP, 9-1-1, and Port Optimization Before VoIP, telephone numbers have corresponded to a specific physical location within a customer premise, which had a one-to-one correlation to a physical location within the telephone company Central Office (CO) known as a wire center. Each wire center has specific geographic coordinates and a corresponding Public Safety Answering Point (PSAP) from which emergency services such as fire, police, and ambulance are dispatched when a citizen dials 9-1-1. Since the “break up” of the monopoly phone companies, many competitors have entered the market. And there may not be enough telephone numbers in their geographical areas of operation without purchasing another large block of phone numbers that may go unused. The solution? Assign virtual phone numbers from other areas that do not correspond to nearby wire centers. This is great for the new VoIP phone company, but this arrangement breaks the traditional pairings of wire centers and PSAPs. Is this a problem? For cell phones, no. Cell phones are mobile and can be anywhere. Cell phones also use a different system to report their whereabouts to emergency services. However, this setup can be a big problem for those who must rely on 9-1-1, such as schools and businesses that may need to summon help. In many cases, new VoIP users, especially residential and small business users, may not realize that emergency services will be dispatched by 9-1-1 from dozens or even hundreds of miles away…until it is too late! Workarounds exist, but 9-1-1 must be tested while setting up new VoIP systems. A county school district in Georgia solved this problem; let’s explore the workarounds they developed. Before the implementation of their new VoIP system, Paulding County School District had purchased their voice services exclusively from their Regional Bell Operating Company (RBOC) BellSouth, now AT&T. As a part of the complete re-evaluation of their network, Paulding County’s Chief Technology Officer (CTO) reviewed alternatives and chose USLEC, an alternative carrier offering lower pricing, better installation lead times, and the full range of services available from AT&T but at better prices. The CTO also found the USLEC account team to be more cooperative in handling deadlines and the exceptions that come up during the cutover of a network his size. And, with seven special T1s – ISDN Primary Rate Interface (PRI) lines, Paulding County was an important customer to USLEC but was very much the “small fry” with AT&T. One problem that was encountered during the cutover was with 9-1-1. As a new telecom carrier, USLEC was assigned a group of numbers to provide to their customers that had previously been geographically associated with Fayetteville, Georgia, some distance away. Ordinarily, the intelligent call routing inside the USLEC network gets all of Paulding County schools’ calls to their proper destinations, but if Paulding County schools dialed 9-1-1, the emergency services would be dispatched by the Fayetteville PSAP and, in many cases, may take 45 minutes or more to arrive at the school. This was clearly not acceptable. The solution that was adopted was both clever and effective. Each school was to have a FAX/9-1-1 line and because AT&T held the numbers that would allow proper dispatch of emergency services from the proper PSAP, Paulding County had to order the FAX/9-1-1 lines from AT&T. And they did. But, Paulding County wanted USLEC to provide all their voice services. Thus, once the lines were installed and running, Paulding County requested that AT&T transfer the service, with the corresponding correct number for 9-11 dispatch, over to USLEC. This is a process known as “porting” and one with which BellSouth is legally bound to comply in less than 30 days from the date of the request. From the standpoint of optimization, special procedures should be put into place to ensure that the numbers going to special phone numbers or lines, such as traditional analog fax or 9-1-1 lines, not be changed, reconfigured, or optimized in any way, including changing the way the calls are routed, their codecs, or other operational characteristics.

178

Chapter 7 Just as the term port has a special meaning in the traditional telephony context so, too, does the term line. A line is also a physical thing and consists of a two-wire or four-wire twisted pair connection between a subscriber premise—a residential or business location—and the telephone company. It is critical that lines be removed, a common term is “retired,” when they are no longer needed because there is a monthly charge per line that does not always magically disappear from the traditional telephony bill as soon as they are retired. In fact, there are auditing services companies and consultants who do no more than audit the lines that are being used versus the lines are being charged and splitting the savings for the first month or so with the client. This is a big and time-consuming task for an organization but one that must be performed by someone. It is also noteworthy that in many cases, especially if a traditional phone company is being replaced by the organization’s internal IPT service or by a competing VoIP service provider, that they are often remiss in their duty of taking items off the bill and often do not allow retroactive credits for items erroneously charged. Auditing and removing unused lines from the phone bill is a critical element in any optimization exercise and one which, in many cases, automated auditing tools can help. In the VoIP context, the terms port and line have their own meaning. In the VoIP context, a port is a physical thing, usually an Ethernet port, to which a VoIP phone or other system is attached. The idea of a line in the vocabulary of IPT is a bit less formal. Generally speaking, from a VoIP point of view, optimization of ports has two aspects. The first is that there be enough physical ports on Ethernet devices such as VLAN switches to accommodate the growing population of Ethernet-connected devices. The second thing to monitor is related to bandwidth: to ensure that each of the ports has sufficient bandwidth to give the call and voice quality needed, especially when aggregated with other traditional data, emerging video, and increasingly common Unified Communications demands. Phone Numbers/Numbering Plan Although the world of IPT is designed to work without traditional phone numbers (instead with alphanumeric Uniform Resource Identifiers), phone numbers will persist until all ties have been cut with traditional telephony and are on IPT-only telephony infrastructures. Experts suggest that that day is a long way off. In the mean time, optimization of phone numbers and auditing of the numbering plan ensures three things: •

There is a sufficient inventory of new telephone numbers to be assigned to new users.

•

Numbers that are un-assigned are returned to the telephone number pool for reassignment.

•

Any internal numbering plan that is in place is being followed. Internal numbering plans are used as often for accounting and cost allocation purposes as they are for geographical number routing.

Each of these steps should occur regularly but should be audited and verified as a part of the optimization exercise. This is another area in which automated tools and log analyzers can assist in the hunt for missing or improperly assigned telephone numbers.

179

Chapter 7

Grooming Grooming is an old term from the fading era of traditional channelized TDM telephony. Grooming refers to ensuring that the least number of T1s and T3s are used in order to optimize costs and network availability. In general, organizations have connected their traditional PBX or other phone systems to the phone company’s Central Office switching equipment using T1s that provide 1 to 24 individual voice channels or Integrated Services Digital Network (ISDN) lines, which are specially formatted T1s that provide 1 to 23 channels usable for subscribers’ voice connections. In today’s IPT world, channelized T1s still exist but, for the most part, connect to softswitches, voice servers, Session Border Controllers (SBCs), and gateways rather than their PBX counterparts. Grooming is an optimization technique that is performed when you have two or more T1s or their larger counterparts, T3s. Figure 7.3 shows the location of T1s or T3s used for telco network access. Each T3, if they are all used, contains 28 T1s for a total of 24 × 28, or 672, voice connections.

Figure 7.3: Traditional telephony interconnection.

The first consideration is redundancy and reliability. If this is a major consideration, you will have pairs of T1s or T3s, each leaving the building through a different exit and, ideally, traveling over a completely different path once they leave the building. It is even possible to have them ultimately connect to two different central offices of the incumbent phone company or two different Points of Presence (POPs) of competing long-distance carriers. This is all to say that during the grooming process, provisions should be made for redundancy and reliability.

180

Chapter 7 Having addressed reliability and redundancy, let’s move to the task of grooming T1s. Each specific channel of a T1 has one or more phone numbers associated with it. These may be individual lines or may be grouped together in a “hunt group” that allows multiple phones to ring in a round-robin cycle before the call is rolled to voicemail or some other method of handling. As the phone numbers associated with the individual channels are moved to VoIP or in some other way released, it is possible to regroup T1s and retire complete T1s. At that point, the billing should cease and further cost savings will be realized. How does this work? Well, let’s say that there are two T1s. One has 24 phone numbers assigned, in the format XXX-YYY-ZZnn where XXX is the area code, YYY is the exchange, and ZZnn represents the subscriber part of the number where nn goes from 01 to 24. Let’s say that the other T1 uses the same setup, but the number is in the form XXX-AAA-ZZnn where AAA is a different exchange. Due to IPT activity and number reassignment, 16 numbers are released on T1 number one and 10 numbers are released on T1 number two. That leaves 24-16, or 8 channels, assigned on T1 number one and 24-10, or 14 channels, on T1 number two. 8 + 14 is only 22, which is less than 24, so all the channels can now be combined on—groomed to—one of the T1s. It is not important that the numbers are not numerically contiguous: the hunt group structure takes care of that, if needed, otherwise it is not an issue. At this point, the T1s have been groomed and optimized and one of them may be released from service. Gateways VoIP gateways, be they standalone or TDM ports on softswitches or VoIP servers, will be connected to the traditional network and usage will, most likely, be increasing. Grooming, therefore, as it relates to gateways and similar systems, will involve increasing capacity to meet demands. There are many ways to calculate the proper sizing for TDM capacity, but the method used by the phone company involves an Erlang calculation of the number of simultaneous channels used for a population of potential callers based upon the length of their calls and their behavior if they can’t make a call. Erlang calculations also involve a determination of the likelihood of blocking of calls using a unit called ‘P’ where P.1 means that 10% (10 in 100) calls don’t get through during the busy hour, P.01 means that 1% (1 in 100) calls don’t get through, and so on. The P calculations are intended to predict a Grade of Service (GoS) during the busy hour, which is the time during which the network will be requested to make the maximum number of simultaneous calls. Optimization of GoS is specifically addressed later in this chapter.

Although the specifics of these calculations are outside the scope of this chapter, it is important to point out that these calculations should have been done at the time of the network design and validation and should be performed periodically for optimization.

TDM Grooming TDM grooming will be a matter of reducing capacity, as described earlier, or entirely removing T1s that are no longer needed because the old equipment is being decommissioned. Optimization includes auditing the bill to ensure that they are removed. 181

Chapter 7 VLAN Capacity The bandwidth and number of simultaneous connections on the VLAN must also be optimized. The demand on the VLAN will be increasing as more and more IPT traffic is moved off the traditional telephone system and over to the IPT system. In addition to bandwidth and simultaneous calls, intermediate devices, in this case VLAN switches, should be monitored to ensure that they have enough memory and processor capability for the increased load. VPN Capacity In conjunction with the internal or external carrier, the capacity of the virtual private WAN network must be optimized. This is yet another area in which automated tools can be very valuable. The specifics of the optimization will be different depending upon the specific underlying technology of the VPN. For instance, a different set of metrics must be optimized if the VPN is based upon MPLS technology than what would be done for a VPN based upon Carrier Ethernet transport. Infrastructure In terms of optimization, the “infrastructure” is considered to be the hardware and software that comprises the network. The transport bandwidth interconnecting the hardware components is considered separately for a variety of reasons, the primary one being that each optimization requires a fundamentally different skill set and set of automation tools.

Hardware Hardware in the IPT sense potentially covers a lot of different components. Consider end-to-end the devices that may be involved and the optimization effort to determine their continued suitability for use in providing the IPT services of your organization. Consider first the wide range of possible hardware devices, the range and diversity of which is only hinted at in Figure 7.4.

182

Chapter 7

Figure 7.4: Infrastructure in transition.

Beginning in the upper left corner is a device that might be a simple traditional telephone. It may also be a more complicated Private Branch eXchange (PBX) phone. That device, be it a mechanical traditional telephone with a keypad and ringer or of the more sophisticated PBX variety, is connected to the network via an Analog Terminal Adapter. The ATA is connected to an Ethernet network, which, in an organization of any size, employs Ethernet VLAN switching and, most likely, traverses the WAN using one or a combination of VPN technologies. Then, there are, potentially, VoIP servers, softswitches, feature servers, gateways, firewalls, SBCs, NAT gateways and other devices. Each component that plays a role in the end-to-end connection must be evaluated and optimized. Repair The optimization is at once both a technical and financial “tune up.” As such, repair procedures must be reviewed for relevance and currency as well as for proficiency of the technicians. It is also important, as a part of the optimization cycle, to audit the repair process to ensure that units that have been repaired are being returned to service.

183

Chapter 7 Obsolescence/Refresh Optimization also involves the decision to decommission certain devices or to refresh the hardware. Specific guidelines should be in place to determine which devices will be decommissioned, or possibly moved to another part of the network, and which will actually be taken out of service. One effective way to implement a decommissioning policy is to take equipment out of service and replace it with its upgraded counterpart when it comes in for repair. This could, dependent upon the component, be less disruptive and less costly, especially for components used by end users, such as IP phones. Managing Surplus Equipment In many cases, especially during a migration from an older technology to a newer technology, there is a great deal of value in the older equipment, especially in a large organization that has an extended timeframe for migration. One of the primary reasons for a migration to a next-generation telephony system—any next-generation system—for many company sites has nothing to do with advanced new capabilities. The real reason for migration is manufacturer discontinuation of older systems and models. When a manufacturer discontinues a product line, availability and rising cost of components for maintenance, repair, and expansion of existing, older, vintage telephony systems becomes a problem. In addition to purchasing upgrades and replacements on the secondary equipment, a manufacturer may market a solution that can extend the lifeline of the older systems and create bottom-line benefits. For example, take equipment that had been decommissioned during the implementation of the new system and send it for refurbishing and placement back in service. This process recycles products either for repair or upgrades, in other locations that are still using the older equipment.

Software Because almost all the components of the IPT infrastructure are, to some extent, computers, software maintenance, enhancement, upgrade, and the management of patches, licenses and security is of the utmost importance and all these areas must be audited routinely. There is no better time to accomplish this task than during the optimization phase. Maintenance Software maintenance is an ongoing, often evolving, process. Optimization of the software maintenance process ensures that all maintenance is performed in the simplest, most secure manner and utilizes as few human and network resources as possible.

184

Chapter 7

Enhancements Enhancements are differentiated from upgrades in that enhancements bring additional functionality and are voluntary. Upgrades are needed to keep software and firmware within a common operating range and they are mandatory. When enhancements are made available, the process should ensure that all users to whom the enhancement apply are notified and given the opportunity to download it. Part of this process should be to ensure that users to whom an enhancement does not apply are not “teased” with knowledge of an enhancement they cannot get unless they are also allowed to change their equipment type, OS, and so on. The optimization process involves looking at enhancements that have been made available, judging user acceptance, and making a determination about whether an enhancement will be supported going forward. It is important to note that the decision to offer, or allow, any enhancement should be made very carefully on the front end because all enhancements have costs—resources such as memory, processing capacity, bandwidth, user training, support—and that once a feature is offered, it is almost impossible to withdraw the offer or to withdraw support for enhancements and features that are already deployed. Upgrades An upgrade is involuntary, so it is often given a lower priority by users, is disruptive, or may change the way a system operates, or impact its ease of use. Optimization of upgrades involves first auditing to ensure that all pertinent upgrades have been made and second to assess the impact of the upgrades. Another factor that should be considered as part of upgrade optimization is licensing. As licenses can be either per end-point or time based an audit should ensure that an organization is not paying for unused licenses or breaching licensing agreements. Patches and Security Audits Periodically, manufacturers release software “band-aids,” small pieces of computer code that are intended as a temporary fix to a problem, most often a security vulnerability. These “band-aids” are called patches. Very often patches fix one problem or close one hole or vulnerability but create others. The assumption will be made here that proper due diligence was applied to the decision to apply or not apply certain patches and that the optimization process will audit to ensure that all appropriate patches are applied and that they are still needed or, if they have been superseded by upgrades, that the patch is no longer being applied and taking resources.

185

Chapter 7

SLA Improvements One of the most important aspects of the initial guidelines and baselines in the SLA was to make the transition from traditional telephony as seamless and easy as possible. Admittedly, this was not really necessary for many of the users as their perception of QoE was forged more as a result of recent emphasis on less-than-reliable “can you hear me now?” cell phones rather than historically superior hard-wired land-line phones. Nevertheless, emphasizing quality of the new telephony services relative to traditional “toll quality” is a safe bet as it will optimize the quality of [user] experience for all users. At this point in the life cycle of the IPT system, the baselines of the new network relative to all of the important SLA metrics—delay, delay variation, packet loss, and availability—are the ones with which the users are familiar day-to-day. The optimization effort should be to improve the metrics but not so quickly as to cause users undue concern. One of the subtleties of “quality” is not that “quality” represents “the best” but rather that to be considered of a high quality, a product or service should be consistent. There is a lot to be said for the confidence one gets from consistency. These observations provide guidelines for the rate of improvement in SLA metrics. Keep in mind that “quality” is relative and that improvement, regardless how well intentioned, represents change and that change, if noticed, always brings concern and sometimes stress and resistance. The following metrics, often tracked in the SLA and compliance process, are among the measurements of the greatest importance and the ones where third-party measurement and reporting tools can have the biggest impact. Beyond their use in the optimization cycle, the regular tracking of these metrics and exception reporting when the associated targets are not met, comprise the “dash board” or “control panel” of any modern multi-media network and its associated services. Delay Delay impacts the user perception of quality in many ways. There are a variety of things that can be done to reduce delay, or the perception of delay, and, therefore, positively impact the user’s QoE and set new bars for future performance. Any call, regardless of whether it is a data call, phone call, video connection, connection oriented, or connectionless, has three distinct phases: call setup, information transfer, and call tear down. There are several points in the process for optimization.

186

Chapter 7

Call Setup On the downside, if call setup takes too long, an impatient user, more familiar with the near instantaneous call setups of the traditional telephone network, might abandon a call and retry or even attempt to place a service order. For this reason, there are certain standards that must be applied. The first element is the time (or delay) to dial tone. If an audible dial tone, or some other indication that a call can be placed, is not presented to the caller within a very short time, the call may be abandoned. Beyond that, once the digits are pressed or the destination number otherwise signaled to the system, the next delay is the call setup. Call setup times can be impacted by a variety of factors that can be optimized. For instance, if call servers are centralized, optimization can improve call setup times by increasing bandwidth between the telephony device and the call server or optimizing the placement of other systems that might contend with the call setup for resources. If, for instance, data servers and telephony servers are on the same VLAN, they can be separated. It is also possible to further divide telephony users into multiple VLANs or to upgrade telephony servers to ensure that they are responding as quickly as possible to call setup requests. During Call Analysis of traffic patterns, endpoint pairs, time of day, and time of month variations and shifts due to time zones will yield a lot of information about the consistency or lack of consistency of performance during a call. For instance, using automated tools to gather and compare delay, delay variation, and loss statistics for all calls and then plotting them graphically will show times when variations are most likely to occur and the causes of variations can be isolated and optimized. This is an important part of any ongoing optimization effort and, if done properly, can spot problems before they are reported by users. Call Tear Down/Release Once the call is completed, the call is released. This is an area of optimization that is most important if calls will be serial, such as in out-bound call centers. If a call is terminated and then the telephony user goes about non-telephony tasks, this delay is buried in the network, invisible to the user and, therefore, far less important to optimize. If, however, one call ends and another call must be made immediately, the call release time becomes important. The good news is that minimizing call setup time and performance during the call will usually have a positive impact on the call tear down delay because the entire end-to-end process has been optimized. Delay Variation Delay variation, also known as jitter, is greatly impacted by other traffic. As with delay minimization discussed earlier, delay variation should be tracked regularly and any variations outside of a set range should be reported and researched immediately, independent of the optimization cycle. But, as a part of the optimization cycle, delay variation must also be analyzed, trends noted, and improvements made. The same process applied for delay minimization will yield improvements in delay variation.

187

Chapter 7 Sample and Packet Loss Sample loss, defined as loss of the actual content representing human speech being transmitted, is one of the network impairments that will have the greatest audible impact on voice quality. If human reports or the results from automated measurement tools indicate that the observed or calculated MOS due to sample loss is outside of established benchmarks, the problem should be addressed through troubleshooting and repair. The optimization process, however, should establish new standards for packet loss. Sample loss and packet loss are very closely related and should be optimized together. Sample loss refers specifically to the loss of samples of human speech that are created at the speaker’s end of the call and must arrive at the listener’s end and be played out as a part of the call. Packet loss refers to the loss of the protocol packets containing the human speech samples. If each packet contains a single speech sample, sample loss and packet loss are one and the same. However, for purposes of bandwidth efficiency, it is common that more than one speech sample be placed in each packet. The first task of optimization, therefore, will be to determine the settings for the network being optimized and to understand what, if anything, can be done to improve the current settings. Research has shown that humans can tolerate a sample loss of up to 10% and in fact this may have been the original benchmark for the network, especially if the IPT system user’s frame of reference for voice quality was cell phones. In any case, the goal of optimization is to analyze the loss, optimize the transport network and its components, and to establish new benchmarks. Availability Availability should be tracked regularly by operations personnel and will usually be shown as a general network availability figure, unless specialty tools are used to monitor availability of IPT as a business service. Basic tools that use simple network functions such as PING will provide sufficiently accurate availability information for IP networks. Although this is fine for most needs, it is insufficient for purposes of optimization of IPT services. For IPT services, availability should go beyond the ability to physically connect to the network and should include the simultaneous availability of all constituent components needed for the end-to-end call, call server, and so on. In other words, the availability of all individual components is important in the context of the availability of the telephony service as was discussed in the section earlier titled Scope of Optimization. Automated tools are very important in assessing service availability and isolating the specific components responsible for any degradation of availability.

188

Chapter 7

Telephony Service Availability Beyond the availability of the IPT network and its IP infrastructure, it is critical to monitor and optimize the availability of the traditional telephone service. This usually means the connections to the PSTN. IP networks and traditional telephone networks are different in many regards, one of the most profound being the way in which each handles offered load. In the case of IP networks, which operate on a “best effort” basis, additional load is accepted and, in the absence of the more sophisticated QoS mechanisms that are just now being implemented in IP networks, connections compete for network resources on an equal basis. Basically, IP allows packets to fight it out among themselves on a level playing field. Telephony networks have specific, clearly identified TDM channels that may be occupied by one and only one call at a time. And, with the exception of certain military and public safety systems, the rule is “first come first served.” The metric within IP networks that must be optimized is Quality of Service (QoS). The metric within the traditional telephony network that must be optimized is Grade of Service (GoS). GoS GoS represents the probability that an attempted call will get through. A typical target used by telephone companies is P.02, which means that only two calls out of every hundred attempted during the busy hour would not go through. Depending upon the number of people who might place calls, the length of the calls they might place, the number of telephone devices, and the person’s behavior if their call does not go through, GoS creates some level of oversubscription. That is to say, GoS measurements let telephone companies provide fewer actual dial tones than there are telephone devices because, except in certain rare cases, all the phones will not be in use simultaneously. This design approach is called a blocking system because there is a statistical probability that under certain circumstances, which are specifically predicted and allowed for in the design, calls may not be connected. Just what is meant by the “busy hour” and the calls attempted during that time is detailed in the following sidebar.

189

Chapter 7

The Busy Hour and BHCAs The “busy hour” is a somewhat mythical and mystical thing referred to by traditional telephony so frequently and so casually that an outsider might draw the conclusion that it actually exists. In fact, there are many busy hours. Because of the natural ebb and flow of the human daily cycle, the first busy hour, both for phone and data, is roughly 10 am to 11 am, plus or minus, local time. This is referred to more specifically as the morning busy hour. The afternoon busy hour occurs at roughly 1:30 to 2:30 pm local time. The evening busy hour occurs at approximately 7 to 8 pm local time. Many things can impact the busy hour. To begin with, busy hours tend to have more calls occurring in a smaller time window if the calls are between points in the same time zone. Busy hour calls between points in adjacent time zones have the effect of having fewer simultaneous calls. Busy hour calls between points in non-adjacent time zones have the effect of yet fewer simultaneous calls. It is also true that the morning and afternoon busy hours will be of greater interest for business telephony systems while residential telephony systems will focus more on the evening busy hour. It is also true that calls between residential and business systems will more likely occur during the morning and afternoon busy hours. Work at home and Small Office /Home Office (SOHO) will also have an impact on call distribution. All of these variations should be taken into account when considering your own busy hour for optimization and planning purposes. Another important consideration is Busy Hour Call Attempts (BHCAs). The number of simultaneous calls traversing a gateway, for instance, can be very large and is limited only by the number of TDM circuits interconnecting the gateway to a PBX, PSTN, or a long-distance carrier and the amount of memory and table space in the gateway. The amount of resources taken for converting IP call packets to TDM bits is small compared with the amount of processing resources for setting up calls and maintaining call state information. For this reason, many times gateways can handle a large number of calls of longer duration than they can shorter calls requiring thousands of call setups per second. Most enterprise gateways, and, in fact, many service provider and carrier-class gateways, are not capable of handing the BHCAs needed for large networks. BHCAs are another very important measurement and optimization number.

If the IPT system being optimized is trending toward more on-net calls, meaning more on IP net calls, then GoS on gateways connecting the IP network to the traditional telephone network will be continually improving because the number of simultaneous calls will be decreasing. In this case, the optimization process will be a financial one: decommissioning lines and optimizing access T1s as previously discussed. However, if growth in IPT users means that more calls are flowing across the gateway between the IP network and the PSTN, the number of traditional circuits interconnecting the two will need to be increased. Blocking/Non Blocking Access Ideally, the number of circuits available between the IP and traditional parts of a network should be such that call attempts are never blocked at the gateway. Non-blocking access as this is called, may be somewhat expensive to provide under normal circumstances but is a default situation under others. For example, if there are 12 total phones on the IP network and the IP network is connected to the PSTN with a T1, the interface is, by default, non-blocking because the T1 allows 24 simultaneous connections and only 12, at most, will ever be needed. However, if the number of phones exceeded 24 and a T1 were used, there would be a potential for calls to be blocked. If for instance, there were 25 phones and each caller wanted to go through the gateway simultaneously and make a call to a destination on the PSTN, one of them would not be allowed through. However, it is likely that someone might not be in the office, that they don’t all need to be on the phone simultaneously, or that one or more people within the office might be calling each other and, therefore, not going across the gateway to the PSTN.

190

Chapter 7 Success Rates of Call Setup IPT gateway systems and their PSTN counterparts on the other side of the connection have the ability to report the success rates of call setups and to distinguish between call attempts and call completions. These are metrics that should be monitored and optimized as they are a key component of the overall IPT QoS delivered to the caller. Erlang B calculations are used to estimate the actual number of interconnect circuits needed, but observed statistics should form the basis of the actual number installed. Related Gateway Issues Not only are circuit connections required between gateways and the PSTN or other TDM devices such as PBXs but also the gateway must be sized to handle the call volumes. BHCAs should be closely monitored and the gateway’s resources should be sufficient to ensure that BHCAs will be maximized and the number of blocked calls will be minimized. It is also important to note the distinct differentiation between calls that are intentionally blocked, and reported as such, due to lack of available circuit connections and those that are lost into the black hole of telephony due to lack of memory, buffers, or processor capacity.

Prices and Costs Cost reduction must certainly be a part of each and every organization’s optimization exercise. To clearly understand the component elements for which budget dollars are extended, this optimization should be divided into hardware costs and service and support costs. Hardware Costs To some extent as a result of Moore’s Law—the observation made in the mid 1960s by Intel cofounder Gordon Moore that products based on computer chips have a predictable increase in capability while their cost goes down—end-user organizations have an expectation that hardware prices will continue to decline. To some extent, this is a realistic expectation, largely because they have and, in the hardware cost category, at least, they should continue to. Keep in mind, however, that declines are percentages of the new cost, not the original cost, so while there will be continuing decreases they will, in actual dollars, be less and less. The specific prediction upon which Moore’s Law is based says that capacity will double and cost will be cut in half approximately every 14 to 18 months. What this means in real terms is that something that cost $100 in the year 2000 will cost $1 in 2010. So what does all this mean in terms of actual hardware costs? In terms of the cost of the raw hardware—purchased as a commodity item in an open market and priced independently from any support services—costs will continue to come down. Larger organizations, who have a lot of clout as a result of their total dollar expenditures, often write a 20% year-over-year decline in prices into multi-year purchase contracts. In fact, the sellers are usually happy to have such contracts in place and are willing to include such clauses to reduce the chances of losing the business to a competitor and to lower their costs of constantly selling to the large account.

191

Chapter 7 Hardware should originally have been chosen to yield the greatest possible Return on Investment (ROI) and lowest Total Cost of Ownership (TCO). In other words, it should originally have been selected not only for the lowest price but also for the longest service life, greatest upgrade potential, optimum ease of use and management, and a variety of related factors. Hardware optimization should first consider the extent to which technology refresh or upgrades could be done using existing equipment before purchases of new hardware are considered. One example of these considerations for an IPT environment is in the choice of user connectivity options. When the IPT service was originally installed, for instance, software upgradeable SIP phones may have been considered and compared with the less expensive option of using a lowcost ATA and keeping the existing traditional analog phone. At the point when that consideration was first made, it is possible that the cost of the software upgradeable SIP phone exceeded $600 each while the cost of using an ATA with the existing phone was $120. Now that it is time to optimize the hardware, it might be prudent to review the earlier decision and consider the impact of time on costs. What you might find is that, due in part to Moore’s Law and in part to competitive and market pressures, SIP phones costing around $100 are now available and, even though ATAs now cost less than $40, it might be wise to begin using SIP phones—even if not a complete replacement, at least for new service and to replace the older ATAs when they come back in for repair. Service and Support Costs Anecdotal evidence tells of organizations that are so aware of the constant decrease in prices of technology that they never make a purchase decision for fear of not getting the best deal. This statement may be a bit extreme, but is really not far from the truth in many cases. As true as this constant reduction of price, with increased functionality, is in hardware, the opposite is true in service and support costs. Service and support costs are based upon the cost of human resources. Savings have been achieved in the past, but service levels have often suffered, by tapping workers in India, Pakistan, Ireland, the Philippines, China, and elsewhere, but even now there is an increase in the price of the best of these resources accompanied by the desire to improve service levels. A qualified VoIP support technician in, India, for instance, might know the technical aspects of the system and be a world-class troubleshooter but if they are unable to clearly communicate verbally with the customer and determine the nature of the problem, service suffers. Service and support costs must be carefully weighed against the actual cost of an outage or poor support. This is one of the areas in which all costs need to be factored in: What are the acquisition costs for the service or support resource? Will it be outsourced or performed in house? Certain aspects of service cannot, for instance, be outsourced to India. Installation, for example, may require a costly truck roll and in-person visit by a qualified, and expensive, technician, but carefully designed self-installation tools might lower the instance of truck rolls, lower costs, use the qualified technicians more wisely, and get services up and running faster. Assuming, of course, that this process does not frustrate the end user and make their job substantially more difficult.

192

Chapter 7

In-Sourcing vs. Outsourcing One last consideration that should always be made, which is a hybrid of hardware and software optimization, is consideration of in-sourcing versus outsourcing. If you are presently outsourcing some or all your IPT infrastructure and/or management, it is possible that the assumptions upon which that decision was made have changed and it is prudent to review the outsourcing decision. If, however, you have chosen in-sourcing, or providing some or all of your own components and support in-house, it is also prudent to review that decision. In many cases, there is a compelling business case for outsourcing that is counter-balanced with a desire to more tightly control the mission-critical IPT services, and tighter control often wins because it is driven by fear of the unknown. After the organization is more familiar with the operation of the IPT services, and the fear of the unknown has become familiarity with the known, the organization is better able, and more comfortable, managing those services through an outside organization. At this point, it may be time to take the path of the more compelling business case and the organization has developed the operational maturity to do so. There is a variety of aspects of the ongoing operation, and in fact the optimization of, an IPT system that could be considered for outsourcing. This is an area in which some creativity and new approaches to running the IPT aspect of the business could pay big dividends.

IP Contact Center Optimization Although all the foregoing optimization techniques and approaches apply to IP Contact Centers (IPCCs), the IPCC is also very different and has its own special needs and requirements. Optimization of the IPCC requires a combination of technical optimization, as discussed in depth earlier in the chapter, and operator and supervisor training to optimize the key metrics upon which the IPCC is measured. For example, the maximum number of calls in the queue, average and maximum call waiting times, and call lengths can all be optimized by properly incenting agents and providing improved agent training and more agents, but there are important dependencies to consider. For instance, providing bonuses for shortest call times or most calls handled will likely provide an incentive for quickly closing calls regardless of whether the problem is solved. To ensure that the callers’ problems are being addressed satisfactorily, a follow-up method is required. A combination of scheduling optimization to ensure that the proper skills are available when they are most needed will be required. Agent performance can be reported on a variety of metrics that vary by call center and applications, but typically involve sorting out and reporting an agent or group of agent’s performance statistics by skill group, geographic coverage, closed sales, caller satisfaction, or any other performance-based characteristic. Statistics that can be optimized include number of calls handled, average time per call, number of aborted calls, agent downtime, revenue per agent, and even Interactive Voice Response (IVR) statistics including the number of calls handled by the IVR that did not require a human agent, number of IVR calls abandoned, number of IVR calls handed off to a human operator, and similar statistics.

193

Chapter 7

Summary This chapter has taken a broad brush at many of the most important areas to consider in optimizing the operations and costs of your IPT services. It has gone deeper into topics that are less implementation-specific in nature, such as hardware and service and support costs. The chapter emphasized that optimization is a cycle: optimize your current network, set new operational baselines, operate your “new and improved” network for a period of time, collect data, and begin the optimization cycle all over again. The next, and final, chapter will look at all the details and loose ends that did not fit comfortably into other chapters but that are nonetheless important considerations in order to have a truly successful implementation of VoIP and IPT in your environment.

194

Chapter 8

Chapter 8: Sweating the Details and Assuring Success In 1926 self-help author Robert Collier wrote “The great successful people of the world have used their imaginations, they think ahead and create their mental picture, and then go to work materializing that picture in all its details, filling in here, adding a little there, altering this a bit and that bit, but steadily building, steadily building.” We will likely find no better theme as we enter the final chapter of this guide. We started by imagining what could be: •

A new world of telephony where dial tone was only the beginning

•

A new world of telephony that provides a cornerstone for the carrier triple play, Unified Communications, and Multimedia, Internet, Communications, and Entertainment (MICE) systems.

•

A new world of productivity and communications enabled by the Internet Protocol, SIP, and related protocols and technologies.

We then filled in the details, planned our approach, vetted our designs, and developed a strategy for continually refreshing our system and improving, enhancing, and optimizing it throughout its, hopefully long, life cycle. All along the way we strove to keep the best parts of the system we are replacing and to add new capabilities as yet undreamed of when that early system was architected and built. In this chapter, we will discuss many of the details required to ensure success in this new system. Some of the points have been covered elsewhere; others have not. We will also provide a comprehensive checklist for the implementation of a modern, packet-based telephony system that can be used to assure that all the critical details, large and small, are covered.

Legal and Regulatory Issues Technology has always been ahead of the law. It is the nature of technology, and of the law. The gun was invented before there were gun laws. The car was invented before there were automobile regulations. Nuclear power plants existed before there were regulatory controls. In each of these cases, the technology existed before the needed legal and regulatory framework was in place. In each case, however, there was also a body of law that could readily be applied to the new situation. Though not an ideal fit and not completely regulating all aspects of the new technology, it existed, nonetheless. For instance, gun laws contain details regulating specific aspects of guns but prior to law regarding murder, self defense and other areas of gun-related behavior had been around for millennia. The same is true of VoIP and IPT. There is a body of telecom regulation still on the books from earlier times, much of which is being used as a stop-gap measure until more specific regulation of packet-based, as opposed to circuit-based, services comes along. Some of the earlier regulation actually appears as though it may be applied to packet-based systems on a more permanent basis. These issues and their impact on the VoIP and IPT initiatives of all enterprises and organizations warrant the greatest consideration to assure the overall success of any packet voice initiative. Or stated in a more negative way, if these aspects of implementation are not considered, they could cause implementation of the new system to be stopped dead in its tracks.

195

Chapter 8 General Considerations The first question to ask might be “Do specific regulations apply to my organization?” A better question, however, might be “Does specific regulation apply to my organization directly or might it in some way impact me indirectly?” The answers to both questions will vary depending upon the role of your organization. If your organization is the end-user organization implementing VoIP or IPT, then one set of regulations may apply to you. If your organization is a service provider offering packet telephony services commercially in a country, to businesses and/or consumers, then your organization may be subject to a different set of regulations. There is also a third situation that may or may not be worthy of consideration: that is, if a person is an individual user of a consumer, or residential VoIP service that is used for business and reimbursed by the company. In most cases, if you are creating and sending your own packets, there is no specific regulation on your activities, though there may be an indirect impact on you if you are using the services of a carrier or service provider and they run afoul of the law and their service is suspended or shutdown, thereby impacting the voice traffic they are carrying for you. The major categories of issues that must be considered are: •

Countrywide bans of VoIP services or service providers

•

Penalties for non-compliance

•

Emergency services

•

Security and wiretapping/call recording

•

Records retention

•

Taxes and fees

Each will be considered in broad terms here. It is your responsibility to research the specific countries in which you will be operating to determine the present state of regulation in that country and to plan accordingly, including continuing to use traditional circuit telephony until regulations change or become clear.

196

Chapter 8

Countrywide Bans of VoIP Services or Service Providers Numerous countries in the Middle East and some in Asia and South America have either fully or partially banned VoIP services. Often they have banned VoIP services from any provider except the registered telephone companies, many of which are fully or partially state-owned Postal, Telephone & Telegraph (PTT) organizations. In some cases, a limited set of outside providers is allowed to operate. The most commonly blocked VoIP service seems to be the one whose technology is most adept at evading being blocked: Skype. The impact on a multi-country enterprise is that in order to provide packet-based services across the entire user base, possibly incorporating suppliers and customers, a patchwork quilt of providers and services may need to be stitched together. Alternatively, a service-neutral Virtual Private Network (VPN) service will need to be employed that operates in all countries that require it and that will transparently and securely move IP-based voice packets. In many cases, this approach will introduce a level of complexity and cost that many organizations are trying to avoid through the adoption of VoIP. And, in some cases, if the VoIP traffic is not completely new traffic but rather traffic that previously moved across traditional telephone lines, its absence will be missed and it might raise a red flag for authorities to determine where their previous telephony network revenue is going. It is worth noting that many governments around the world view the settlements on their traditional telephony services as an important source of hard currency. Penalties for Non-Compliance There are instances where penalties ranging from fines to incarceration have been levied for both private use of VoIP and the offering of VoIP as a service to the public. It is very important to understand the penalties that are possible and to err on the side of compliance. One does not need to go as far as Namibia or Vietnam to find examples of penalties: there are plenty in the U.S. that have had an indirect impact on companies using the services of certain providers, including services intended as residential services that are used for small office/home office connectivity and have caused service interruptions, delays in bringing services to new areas, and increases in the cost, and therefore price, for the service.

197

Chapter 8

Emergency Services Most industrialized nations have a system for summoning police, fire, and emergency medical services with a single nationwide number, such as 911 in the U.S. and Canada, and 999 in the United Kingdom. On the private enterprise side, it is important to assure that key information such as the location of the party calling for help—for instance, their office building, floor and area on the floor—be transmitted to emergency call centers as opposed to the billing location or the location where the voice switch is located. It is also important for all employees and any other persons in a facility to know exactly how to dial to emergency services. Should they dial 91-1? Or 9 for an outside line and then 9-1-1? The legal liabilities are beyond the scope of this chapter, but this is a very serious area in its own right that deserves full consideration, implementation, and testing prior to actual release of VoIP or IPT systems for general use. The law has generally found companies liable for non-working or inaccurate emergency calling systems. It is also worth noting that most companies’ present systems also are not set up properly. Companies often do not think about this until the system is pressed into service and it does not work. Many companies also rely on cell phones for such services, which have their own set of problems and issues, especially inside buildings. From the standpoint of service providers, there are country-by-country regulations on their obligations to provide emergency services. Companies should ensure that any service providers used fully comply with all regulations and the systems should be tested. Security and Wiretapping/Call Recording Security and wiretapping/call recording are issues that have different dimensions if they are considered in an enterprise or service provider environment. In the enterprise environment, the enterprise has a need to monitor and assess what their employees and/or contractors are doing, the nature of their communications and the applicability of organizational policy and applicable law to those activities. Certain industries, such as the financial industry, also have particular requirements to record calls to verify certain transactions and not just “for training purposes.” For the service provider, call recording reaches into the realm of “legal intercept” referred to commonly as wiretapping. Different countries have different regulations but generally there is a trend to apply traditional wiretapping laws to packet-based services. In general, service providers must comply and may not inform the subscribers of the ongoing investigation. This is especially true in organized crime and terrorism cases. In the United States, the appropriate law is the Communications Assistance to Law Enforcement Act (CALEA), which has recently been expanded to include packet-based services.

198

Chapter 8

Records Retention As with Instant Messages and emails there is an increasingly strong preference both for and against retention of voice packet traffic. On the “for” side are certain regulations, such as certain provisions of the Sarbanes-Oxley law and SEC regulations in the US, that currently require retention of IM and email and may be broadened to include voice packets. On the “against” side is an increasing number of cases in which corporate or government wrong-doing has been able to be proven by the use of records which, if they did not exist, could not have been used for this purpose. Retention of voice packets is an area of current legal interest and one that will continue to be refined in the future. It is also an area that must be considered in any corporate policy. Taxes and Fees One of the biggest benefits of a move from traditional telephony to VoIP was that VoIP was free of much of the regulation and nearly all of the fees traditionally levied against phone companies and the services they provide. This was translated into lower prices and was responsible for much of the initial interest in VoIP. As governments around the world become aware of VoIP as a source of revenue, from the local government to the national level, taxes and fees are being dreamed up for VoIP. Absent strong precedent many taxing authorities may, at least for some initial period of time, be able to levy unbelievably high fees, such as the city of Baltimore has done, with a $3.50 per number tax on VoIP subscribers. This will impact service providers directly and enterprises indirectly due to the service providers passing on the costs of fees and compliance. Country-Specific Regulatory Issues It is imperative that you check current regulations in every country in which you operate to assure that you are meeting all legal and regulatory requirements. In addition to national requirements there may be state or provincial requirements and even requirements imposed by local governments that can harm your packet-based voice initiative. Contractual Issues Many organizations, especially large ones, although the same may be true for small to medium ones with especially high call volumes to specific countries, have specific contracts, often called tariffs with the previously popular “Tariff 12” being a good example - in place with traditional telephony carriers. The tariffs are filed with the FCC or other applicable authority in other countries than the US and spell out the pricing and contract conditions, and often have minimum numbers of minutes that must be paid for whether delivered by the carrier or not. Before calculating the “savings” which can be realized by a move to non-traditional telephony organizations should carefully review and understand the conditions of the current contracts. In many cases there are guaranteed minimum numbers of minutes, or payments, or both. Moving minutes to a VPN or another telephony service, even of the same carrier or service provider, may not satisfy the minimum service requirements and payment must be made for the minimum traffic levels whether the traffic is transported by the carrier or not. Many organizations have been on the path to implementing voice packet services and have found themselves in the position of having to delay implementation until the end of a multiyear contract for traditional telephony services. 199

Chapter 8 Patent Issues It is said that history repeats itself and so it does more often than we might realize. When statements are made that VoIP and IPT technologies are “mature” and “ready for prime time” one must look no further than the early days of traditional telephony to see evidence that, in fact, VoIP and IPT are in their earliest days. If one were to read the newspapers from the ‘70s, that is to say the 1870s, not the 1970s, one would find that patent disputes are common in the early days of the commercialization of any new technology and packet telephony is no different. The 1870s saw bitter legal battles between competitors for the claims to telephony patents. There are many examples just in the lawsuits against Alexander Graham Bell from a variety of individuals including Antonio Meucci and Elisha Gray and the problems Bell had due to his patents being filed in America before the were filed in the UK. History is repeating itself and this is why the final topic in this section is the business impact of the increasing reliance on patent filings and the enforcement of patents as a business strategy on VoIP, IPT and related Unified Communications projects. Once the dust settled in the telephone industry, solid patents and standards became the bedrock of reliable, interoperable global phone systems. All things considered there were really very few variations in phone systems globally, compared with their data counterparts and almost none whatsoever considering the number of variations that could exist. That having been said, there are still hundreds of important variations from country to country and beyond the national variations there are even manufacturer-specific variations. But, even with all of the variations on the themes underlying issues of patent protection, once settled, became a non-issue throughout the remainder of the life-cycle of the traditional telephony system. Because VoIP, IPT, Unified Communications, Internet Multimedia Subsystem (IMS) and related systems are in their commercial infancy, patent issues remain an important consideration for any organization implementing VoIP and IPT systems and/or services. In fact, these are the two categories that must be considered: systems and services. If a company is purchasing systems then they must be cognizant of the fact that the systems they are purchasing, and the underlying hardware and/or software, even if purchased from a historically reliable manufacturer may contain technologies that might later be proven to be infringing on patents that the manufacturer does not own or have license to use. Likewise, when purchasing services the purchaser must be aware that the services may be based upon patents that the service provider does not own or have license to use. What is the “bottom line”? The bottom line is that while most legal outcomes do not require that technology purchased be given back they do make acquisition of subsequent products more expensive or impossible and may cause the suspension of services. While there is really no analysis or report on what the risks are this is still an area that must be considered and, more than anything else, will impact the decision of companies to adopt VoIP and IPT until many of the early patent issues have been decided. The alternative? Rely on extending current technologies and only using VoIP and IPT technologies where they are strategically vital in the mean time.

200

Chapter 8

Technical Issues Even though many technical issues have been addressed in other areas of this eBook they are being re-emphasized here because they often get lost in the background noise but have confounded a proper implementation of VoIP and IPT in many organizations. These topics are presented in no particular order of importance and if properly addressed during the implementation phase none will be particularly thorny but if left unaddressed can become very serious impediments to productivity. Voice Band Data There is a general expectation that telephony systems can handle a variety of Voice Band Data (VBD) options such as: •

FAX

•

Modems

•

Dual Tone Multi-Frequency (DTMF, also known as touch tone and used to access voicemail, bank balances, flight information and other services)

•

TTY/TDD devices for the deaf

•

Alarm systems

•

Other telemetry systems

There is, however, no set of standards for supporting these capabilities within VoIP and IPT systems and each manufacturer and service provider will offer their own set of options. If an organization implements VoIP using Ethernet-based devices, and a standard Ethernet physical interface such as the ubiquitous eight wire RJ45 modular connector then the expectation of VBD support usually goes away right along with the traditional phone jack. This is because the Ethernet physical connector is different from the four or two wire phone jack and there is no place to plug in the fax, modem, TTY terminal or alarm system. If, on the other hand, Internet Protocol Telephony is implemented using some sort of analog terminal adapter or gateway that connects via a standard phone plug then the expectation that the system can handle VBD usually still exists. If VBD is to be handled properly in the new system then proper arrangements need to be made. If the new system is not intended to provide support or VBD then proper training and support needs to be provided to deal with the lack of the aforementioned capabilities or to provide workarounds. If, for instance, faxes are to be supported then the new system must either use 64Kbps Pulse Code Modulation (PCM) or, possibly, 32Kbps ADPCM voice coding which requires additional bandwidth for all calls. If a lower bit rate codec is to be used for normal voice calls then the system must be capable of detecting the presence of the fax signal and switching over to PCM or ADPCM or employing FAX T.38 forwarding, fax servers or similar capabilities.

201

Chapter 8

Numbering Plans VoIP and IPT do not use numbering plans or phone numbers, per se, but they can, and do, have the capability of mimicking phone numbers, at least for an interim period of time. Many in the IP standards bodies have been chiding the current purveyors of VoIP services claiming that they don’t provider “real VoIP” or “real SIP implementation” because they still use phone numbers. The use of phone numbers, at least in an alphanumeric form as part of a SIP User Resource Identifier (URI), will be critical until the migration is made from the older telephony networks and, perhaps, that older telephony network may still be needed for some time to come for its ability to locate subscribers and route calls to them as this is a scalability and interoperability capability as yet to be proven by SIP networks. Voice VPNs Many organizations, from multi-branch single-city banks to multinational manufacturers and consulting firms have long ago established voice Virtual Private Networks (VPNs) with their own specialized dialing plans and associated volume pricing agreements. The particulars of the use of the specialized dialing plans has become so engrained in these organizations that any productivity gains associated with a shift to packet telephony would be more than offset for many years to come by making changes to the dialing plans. Employees know each other’s extension numbers: clients and suppliers know Direct Inward Dialing (DID) numbers based upon those extensions, employees know specific internal numbers such as Human Resources and Security, directories are published, phone numbers are part of documentation and often posted on telephones, bulletin boards and web sites. In short, the entire ability of a company to communicate via voice, both internally and with the outside world, is based upon previously established phone numbers and associated dialing plans. For these reasons it is imperative that the prior dialing plan and associated phone numbers are maintained. It is also imperative, to the extent possible, that the special characteristics associated with the old system, such as the exact sound of inside and outside dial tone, the delay that occurs after dialing the digit (0? 8? 9? 7?) used to get an outside line and all other characteristics be kept unchanged. E.164 Compliance E.164 is the global ITU (http://www.itu.org) standard for telephone numbers. E.164 provides an international numbering plan for public telephone systems in which each assigned number contains a country code (CC), a national destination code (NDC), and a subscriber number (SN). There can be up to 15 digits in an E.164 number and each E.164 number is unique worldwide. With up to 15 digits possible in a number, there are 100 trillion possible E.164 phone numbers, more than 10,000 for every human being on earth. This makes it possible, in theory, to direct-dial from any conventional phone to any other conventional phone in the world by inputting no more than 15 single digits. The hierarchical nature of E.164 as well as the number and use of digits is based largely on work done in the 1880s and 1890s by Alexander Graham Bell and outlined in his famous letter to the board of directors of the Bell Telephone Company in 1894. This same letter described the use of telephone exchanges and foretold of automated switching, something for which Dr. Bell could see the need but could neither describe nor produce.

202

Chapter 8 Maintaining strict compliance with E.164 standards globally at every gateway between the packet world and the circuit world is critical in maintaining the bridge between the two, and assuring interoperability as long as the bridge between the two worlds of telephony is still needed, which is to say, well into the foreseeable future. URIs/URLs The Internet uses Uniform Resource Locators (URLs) such as www.prognosis.com to locate hosts that have some sort of geographic alignment. SIP uses a very similar addressing construct, the Uniform Resource Identifier (URI). A SIP URI identifies a communications resource. Like all URIs, SIP URIs may be placed in web pages, email messages, or printed literature. They contain sufficient information to initiate and maintain a communication session with the resource. SIP Uniform Resource Identifiers The "sip:" scheme follows the guidelines in Internet Engineering Task Force (IETC) Request for Comment (RFC) 2396 - Uniform Resource Identifiers (URI): Generic Syntax. They use a form similar to the mailto URL, allowing the specification of SIP request-header fields and the SIP message-body. This makes it possible to specify the subject, media type, or urgency of sessions initiated by using a URI on a Web page or in an email message. In general, the form of a SIP URI, is sip:user:password@host:port;uri-parameters?headers ●

In a SIP URI user is the identifier of a particular resource at the host being addressed. The term "host" in this context frequently refers to a domain.

●

The "userinfo" of a URI consists of this user field, the password field

●

The @ sign following the userinfo fields. The userinfo part of a URI is optional and may be absent when the destination host does not have a notion of users or when the host itself is the resource being identified.

●

If the @ sign is present in a SIP URI, the user field must not be empty.

●

If the host being addressed can process telephone numbers, for instance it is an Internet telephony gateway, a telephone subscriber field may be used to populate the user field.”

●

Password is optional and is a password associated with the user. While the SIP URI syntax allows this field to be present, its use is optional and not recommended, because the passing of authentication information in clear text (such as URIs) has proven to be a security risk in almost every case where it has been used.

●

The host field provides the identifier for the SIP resource. The host part contains either a fullyqualified domain name or numeric IPv4 or IPv6 address. Using the fully-qualified domain name form, such as phn.company.com, is recommended whenever possible.

●

In addition to the host name a port field may be provided and, if provided, designates the port number where the request is to be sent.

If a telephone number is to be transmitted it is converted into a SIP URI of the form: sip:nnnnn@domain;user=phone or sip:nnnnn@host:5060;user=phone ●

The "user=phone" parameter is a hint that the part to the left of the '@' sign is actually a phone number, in case there are SIP users whose names happen to consist of all digits. Typically a proxy server for the domain will use a dial plan to resolve this into a real destination.

203

Chapter 8 ENUM and Other Numbering Issues The ITU and the Internet Engineering Task Force (IETF) are currently working on a plan called ENUM that will expand E.164 to encompass both traditional analog phones and digital devices, including computers and other devices on the Internet. All types of communications devices— including analog phones and fax machines, digital phones and fax machines, wireless (cellular) phones, pagers, digital modems, digital video terminals, and VoIP devices -- will have unique E.164 addresses with direct dialing possible from any device to any other. Considerations for ENUM and extensions to corporate directories using Light weight Directory Access Protocol (LDAP) and Domain Name Service (DNS) systems must be considered as a part of any comprehensive implementation. Interworking Issues with IPT Service Providers Invariably a packet telephony project of any size will need to connect to IP Telephony service providers for either connectivity between geographically distant internal locations, often including small office/home office sites, or customer or supplier sites but usually all of the above. There are several options and issues that must be considered, all of which have analogs in the earlier phone system. The New Demark The issue of where the responsibility of the “phone company” ends and the responsibility of the customer begins has been the subject of very specific regulation over the years and the issue is very clear. The point at which the “hand off” occurs and responsibility shifts is called the point of demarcation, or “demark”. With individual voice circuits the point of demarcation is marked at a wire connecting a punch-down block owned by the phone company (normally attached to a piece of plywood on the user premise) to another punch-down block (on the same or a different piece of plywood on the user premise). In the case of T1 systems the demark is a phone company or customer owned T1 DSU/CSU device. There are other examples, but the point is that the stage (or place) where the hand-off occurs is very specific in traditional systems. In emerging VoIP and IPT systems the point of demarcation is usually as specific because it is the same carrier, or type of carrier, providing the wires but often a different service provider using those wires to provide the service. This is why a fresh interest must be taken in clearly defining customer responsibilities and service provider responsibilities.

204

Chapter 8

End-to-End Service Management

An increasingly important issue that impacts all aspects of multimedia, whether the service is provided by an internal or external organization, is “end-to-end” service management. In the past “end-to-end” has been defined more as provider edge to provider edge, not including the local access links on the end, unless equipment was co-located in a service provider Point-of-Presence (POP) or carrier Central Office (CO). That definition was expanded by managed service providers to include the local access links and the managed customer premise equipment, such as the router, but did not include the actual service that used the underlying connections nor did it include things such as servers and client end-systems. VoIP and IPT services should be managed - and key metrics such as Quality of Service (QoS) and Quality of [user] Experience (QoE) measured—from the user phone device on one end to the user phone device on the other end. Monitoring, measurement and management of all intermediate pieces from which the end-to-end call is created remains very important but it is the actual service itself that should be measured and managed. It is also critical to view VoIP and IPT traffic in its context as part of the bigger picture of multimedia traffic. Tasks Even back when IP networks were “data only” there were many “flavors” of data - browser traffic, file transfer protocol, terminal emulation, email—all of which have their own performance requirements Assessing IP performance has become even more complex with the addition of two more distinct types of traffic, voice and video, to the multimedia mix. All of the tasks required of today’s IP multimedia networks—functions referred to as applications in an environment often called the single network voice, data, video “triple play”—are easier to deal with if we place them into one of three “loss” categories and cross reference those with two delay-related categories, understanding that these categories are relative to each other and not to some standard baseline. It is in the dynamic prioritization of each type of traffic in the network where we achieve true quality of service that matches, or doesn’t match, the needs of each type of traffic. As shown in Table 8.1 applications on a multimedia network can be broadly divided into those that are less sensitive to delay and delay variation and those that are more sensitive. We can further divide those two categories into low, medium and high in terms of the loss that is relatively acceptable for each application. If we were considering the underlying protocols we would actually place the applications into only two categories, those that use the reliable, errorchecking, retransmit-on-loss connection-oriented TCP protocol or the connection-less User Datagram Protocol (UDP), often referred to by this author as the “Unreliable Datagram Protocol”. In this case there would be no loss in the TCP examples, but possible loss in the UDP examples, but we are looking at this from the user’s perspective and will view the “Acceptable Loss” based on its impact on the total user experience, often referred to as QoE, or Quality of Experience.

205

Chapter 8

Table 8.1: Acceptable Multimedia Loss vs. Delay.

These relationships also imply a certain prioritization relative to Quality of Experience (QoE) which comes into play if packets are competing for network resources. For instance, from the table we can rightly imply that if an email packet were competing with a real-time voice (VoIP) packet for network resources that the VoIP packet would get the resources. Likewise if a device such as a router had to discard a packet and a VoIP packet were competing with a browser packet that the browser packet would win because the discarding of a browser packet would cause the retransmission of the packet, placing a further demand on the network to do the retransmission. Quite the opposite is true of voice. Voice will sustain a 3-5% loss before the quality drop is noticeable and can randomly lose up to around 10% of voice samples before intelligibility begins to decline substantially. Considering these dynamic trade-offs, based upon the demand on the network at a specific moment in time, is an important a part of assessing the performance of your multimedia IP network. If this explanation seems to run counter to traditional thinking, based upon the well-known characteristics of IP, TCP and UDP you are quite right. The underlying protocols were designed for a uni-media, not multi-media, world and the shift in network operations is a function of changes in the software code in routers and switches that lie in the path between one communicating system and another, not in a fundamental re-write of TCP or UDP. Tools A specific, finite list of characteristics may be ascribed to certain types of packets either as they are formed in their respective end systems, as they enter the network, or at any other point where they are handled by systems capable of such discrimination and traffic marking. Each of the combination of characteristics can be matched to a specific designation, or class of service, with the intention that traffic so-marked will achieve the corresponding promised quality of service. Any given network may or may not implement all of the classes of service or may have some special hybrid. Constant Bit Rate (CBR) promises service most like a dedicated circuit. It promises a constant bit rate and near zero loss. Real-Time (RT) and near-Real-Time (nRT) are two flavors of Variable Bit Rate (VBR) classes of service that promise performance below that of a guaranteed circuit but better than the best effort objective of historical IP networks. Available Bit Rate (ABR), which promises a modest “best effort” service with a minimal assured throughput, even in times of congestion, and Unspecified Bit Rate (UBR), which makes no guarantees whatsoever, in most cases, round out the classes of service. By aligning the needs of each of the applications to specific classes of service it is possible to provide the full range of tools needed to provide the mixed quality-of-service needed for multimedia traffic and to request the network to provide for the specific needs of each.

206

Chapter 8 Differentiation and Outcomes The key to success is to properly differentiate and then to aim for the best but plan for the worst. In today’s multi-terabit networks it is easy to picture an optical highway system so empty that every packet can receive first class treatment. While this may be an accurate high level view it neglects two important facts: the first is that one must actually be able to get on the highway and the second is that regardless of the low and declining cost per bit per second, businesses are always driven to save money and often sacrifice performance to do so. In this frame of reference it is easy to see that trade-offs are made in how the carrier or service provider’s bandwidth is carved up and how it is paid for: regardless of low price there is still a cost. The key is to properly differentiate traffic and not to over-provision or under-provision the system relative to the need. Table 8.2 shows which class of service is ordinarily applied to each sample application. In a perfect world—and it does happen in the networking world, especially at off-peak times—all traffic is marked according to what class of service it is promised, but all packets, regardless of marking, traverse the network in near-wire-speed times and none of them are delayed or discarded. In that case the differentiation and class-of-service marking of packets is inconsequential and it could even be argued adds to delay and overhead. So, why do it? Because the network is not a perfect place, especially at peak times, and the marking tells the network how to resolve conflicts for resources when they ultimately arise in the shared environment of the IP network.

Table 8.2: Class of Service (CoS) Examples.

207

Chapter 8 Monitoring and Testing

Each of the classes of service will have some associated metrics that can be measured and interpreted relative to its impact on the application. The metrics that are normally gathered and used are packet loss, delay, delay variation (jitter) and service availability. Constant Bit Rate, for instance, has a packet loss near zero, delay near wire speed (meaning that distance is a more important factor than the performance of intermediate pieces of equipment), delay variation near zero and a 99.99+% availability. On the other end of the spectrum is Unspecified Bit Rate, with no guarantees at all, in most cases. To judge both is easy, and fair. A packet loss of a whopping 2.6% for CBR is considered to be abysmal performance and most likely subject to a financial penalty against the carrier. A loss of only 2.6%, especially during peak traffic times, is probably pretty good for Unspecified Bit Rate. The key is to have the target metrics, and penalties, spelled out in advance and to monitor performance relative to the agreed upon targets. This is best done in the Service Level Agreement and monitored via a combined program of intrusive testing and passive monitoring. SIP-T Traditional telephone networks, as shown in Figure 1, employ two distinct parallel networks to create, facilitate and delete end-to-end telephone connections. The network interconnecting the ISDN phone on the left via discrete circuits running through the grey cloud in the middle to the analog phone on the right is only the network that carries the bits between the two phones.

Figure 8.1: Traditional telephony architecture.

The blue cloud in the back, in this case marked “SS7 Network” because it facilitates the signaling system used in North America, is a parallel control network comprised of highly reliable dual-home packet switches that does the set-up and tear-down of the circuits. Without the secondary signaling network there would be no calls. This is quite different in IP-based networks. IP-based networks are characterized by the fact that the same network, the IP-based Internet, intranets or VPNs, carries both the content and the signaling traffic. One aspect of IP networks that has concerned telephony engineers from the beginning, is that while their SS7 networks provide highly reliable out-of-band signaling to insure the integrity of the network and minimize risks to network reliability and, therefore revenue, IP networks started out as “best effort” with control and user traffic competing for shared resources on an even footing. This, fortunately, has changed for a variety of reasons and control traffic can receive priority treatment.

208

Chapter 8 It is important to consider the signaling needs of the network during design and implementation. The Session Initiation Protocol (SIP) is the most likely choice for voice sessions in the network and, therefore, SIP-T, Session Initiation Protocol for Telephones, will be the most likely choice for signaling, call set-up and tear-down. SIP-T provides a framework for the integration of legacy telephony signaling into SIP messages. SIP-T accomplishes this using two techniques known as ‘encapsulation' and 'translation'. At a SIP gateway, SS7 messages are encapsulated within SIP in order that information necessary for services is not discarded in the SIP request. However, intermediaries like proxy servers that make routing decisions for SIP requests cannot be expected to understand SS7’s particular signaling protocol called the ISDN-User Part (ISUP), so simultaneously, some critical information is translated from an ISUP message into the corresponding SIP headers in order to determine how the SIP request will be routed. While pure SIP has all the requisite instruments for the establishment and termination of calls, it does not have any baseline mechanism to carry any mid-call information (such as the ISUP INFormation/Information Requst (INF/INR) query) along the SIP signaling path during the session. This mid-call information does not result in any change in the state of SIP calls or the parameters of the sessions that SIP initiates. A provision to transmit such optional applicationlayer information is also needed. Electrical Power AC wall power or Power over Ethernet (PoE) must be provided to power most IP phones. In many cases consideration of power is not a part of network design and results in substantial additional costs and implementation delays.

Business Issues Are packet-based telephony systems less expensive than their circuit-based counterparts? Is it possible to tell? Are the systems so different that it becomes an apples vs. watermelon comparison? It is most likely to be true that IPT systems, which use a new engine to deliver traditional voice services will be less expensive, possibly 30 to 60% less expensive, than their traditional telephony counterparts. Most organizations, however, eschew a strict replacement approach for one of replacing dial tone and then adding new capabilities, placing them squarely in the VoIP realm, regardless of their choice of underlying technology. With VoIP the organization adds additional capability but also brings additional training requirements and budgetary and staffing needs.

209

Chapter 8

Budgets Packet-based telephony systems are different from their circuit counter-parts in many important ways that have an impact on budgets. For instance, hardware becomes obsolete much more quickly and refresh or replacement must be budgeted. In the old telephony networks it was possible to have a standard touch-tone phone in service for a decade or two. Not so with today’s IP phones, palm devices and wireless phones: the technologies and capabilities are changing too rapidly. A reasonable refresh cycle is twelve to eighteen months depending on your industry. Software and services are also areas that are quite different than in the old telephony days. IP phones often have software licenses or costs for specific features that were either non-existent or associated with the phone switch or PBX in traditional systems. These items must be managed and budgeted. Staff One of the biggest misconceptions in the early days of IP telephony was that voice was “just another IP application” and that it could be managed successfully by the existing staff with existing tools. That is a misconception that is now understood to be wildly wrong. What many organizations are now coming to grips with is that they must keep at least some of the former telephony staff members. Ideally from this process a “multi-tech” with the requisite skills to handle voice, data and video issues will emerge but wide-spread availability of such super-techs is still some way off. Organizations should guard against skills attrition in their staffing efforts and assure that they are providing both training of new staff and cross-training of existing staff in all aspects of voice, data and video. Repositioning/Redeployment of Old Equipment While some parts of the organization may be moving rapidly to deploy packet technology it may be difficult or impossible for other areas to do so and it might make very good sense to reposition or redeploy equipment that is being removed from one area of the organization to another area. What if one division has chosen not to make a move to packet telephony at this time? They may still need to upgrade their old key system to handle more phone lines. Or, what if one location is in an area where regulations forbid VoIP or there is a minimum number of minutes required on a tariff that can’t be met if traffic is shifted to VoIP before the end of the contract period? These are great places to reposition older equipment. Secondary Market for Old Equipment For a variety of reasons, the most important of which is manufacturer discontinuation of specific lines of traditional telephony equipment there is a thriving secondary market for older telephony equipment. The secondary market should always be considered vs discarding older equipment.

210

Chapter 8

Donation of Old Equipment In the early days of packet telephony, when the direction was less certain, donation of old equipment to educational or non-profit organizations for their own internal use or training purposes made sense and the donations were welcome. The farther we go toward world packet domination the less true this is and the more that donations of this type are seen as liabilities. It is still possible to donate old equipment but the context needs to change: donations to manufacturers, carriers or other museums are becoming more important, especially for pieces of equipment that are particularly unique or have an interesting role they played in a historical event. Organizations should not underestimate the value of their old, outdated equipment and should always look at the value of donation of that equipment as opposed to discarding the equipment. Revisiting TCO and ROI Models for your organization In Chapter 1 we described several different approaches to determining Total Cost of Ownership (TCO) and Return on Investment (RoI) models. Your approach might exactly parallel one of the suggested methodologies, might closely resemble one or two or might be completely different. In any case once the new system has been installed and stabilized, and once a sufficiently large number of users have been using the system to get some meaningful numbers it is time to revisit your TCO and RoI models and see how you’ve done. Is the Total Cost of Ownership lower or higher than originally estimated? Was the investment returned in the time frame estimated or does it appear as though it will be? Will it take longer? Less time? Why or why not? Many organizations never go back and recheck the initial assumptions but it is important to do so. The first reason might be for “bragging rights” to a properly performed prognostication or, in any case, maybe some original assumptions were inaccurate or never made it into the final design. Regardless, it is a valuable and important exercise to revisit the original planning exercises and clear up any discrepancies.

211

Chapter 8

Security, Privacy, and Safety Generally speaking because VoIP and IPT operate over IP networks they are subject to the same security vulnerabilities and issues as any IP service. However, to the extent that the calls may traverse traditional phone systems and networks they are subject to the security vulnerabilities and issues of traditional telephone networks as well. For this reason a plan must be put into place to assure the integrity, security and privacy of all phone calls regardless of the system over which they are connected. This would also be a good place to discuss wireless vs. wireline issues as well. A complete and comprehensive security analysis at each design step followed by security assessments of the actual system once it is installed and periodically thereafter are critical to the operation of the system. The four areas of security that will need to be addressed are: •

Privacy

•

Operational continuity

•

System integrity

•

Theft of service

Privacy assures that information being transmitted remains between the communicating parties and cannot be “overheard” by any third party. Privacy can be accomplished through a variety of means, the most effective of which is end-to-end encryption. End-to-end encryption scrambles the signal at one end-user device and unscrambles it at the other end. End-to-end encryption assures that the message is as secure as possible from one end to the other. Operational continuity is the process of protecting a system from issues such as Denial of Service (DoS) attacks and assuring that calls can be processed under any foreseeable circumstance. Denial of Service attacks that are beginning to emerge in VoIP and IPT systems include restarting call servers so that the traffic generated by all of the connections attempting to re-establish themselves brings the system down. System Integrity problems include situations where hackers can masquerade as internal callers or spoof IDs and phone numbers as well as assuring the integrity of call logs and other tools that can be used after-the-fact for their forensic value. Theft of service, often called ‘toll fraud” in the traditional phone network has almost disappeared as an issue from the lists of most managers of packet voice systems but shouldn’t. In the past the number one impact of toll fraud was the high cost. The cost of stolen minutes had to be paid to the telephone company and this had a direct financial impact. Packet-based systems are so inexpensive, by comparison, and are often flat-rate so the financial impact is not so noticeable. Theft of service, or unauthorized use by employees, steals time from the job and often has other implications such as setting up an organization for prosecution if their systems are being used by hackers, criminals or terrorists.

212

Chapter 8

Emergency Calls—911, 999, 112, and So On In addition to the regulatory issues discussed earlier it is imperative that an organization fully understand the liabilities associated with not providing proper emergency calling. Not only are the lives of employees, contractors and visitors placed at risk with a non-functional emergency calling system property can be placed at risk if it is not possible to summon fire fighters in case of a fire in the building. As stated earlier it is imperative that emergency calling procedures are well documented and tested to assure they will operate properly when needed and summon assistance to the proper location.

Telephony Migration Checklist The telephony migration checklist provided at the end of this chapter offers a single list that can be used to double check all of the key aspects of your migration from traditional telephony to IPbased communications.

Summary The successful implementation of VoIP and IP Telephony is a topic that is broad and deep all at once. In this eight chapter e-book we have made every attempt to touch on all of the issues that you might encounter and to delve deeper into those that are likely to be of greatest importance. We have provided templates and checklists, guidance and “war stories”. With all of this input and your own experiences to guide you it is still impossible to get everything exactly right on the first go but with careful planning and implementation it will be possible to get closer to perfection than would ever have been possible by just “taking a stab at it”. If we have provided some valuable insights, if we have shown some possible booby-traps and some practical work-arounds and saved you time, aggravation and money we have accomplished what we set out to accomplish at the beginning. Best of luck to you as you pioneer a new way of communicating and find the best path forward for your organization and its users.

213

Chapter 8

Telephony Migration Checklist Justification _ Return on Investment (RoI) _ Return on Effort (RoE) _ Total Cost of Ownership _ Strategic capabilities _ Manufacturer discontinuation of older systems Expectation Management _ Users _ Different is not bad _ Provides all important functions of the current system _ Will provide important new capabilities. _ Management _ Total CAPEX cost may be the same or higher _ Total OPEX cost may be the same or higher _ Must assure the continuity of voice communications _ Acquire advanced capabilities of strategic benefit _ VoIP is not “free voice” _ VoIP is not “just another application.” Budgeting and Planning _ Reuse/Redeployment of existing assets _ Acquisition _ Products _ Services _ Managed services and outsourcing _ Support _ Internal Help Desk _External Help Desk Service Level Agreements (SLA) _ Parameters _ Availability _ QoS: Packet loss _ QoS: Delay _ QoS: Delay Variation _ QoE: MOS _ QoE: E-Model/R value _ QoE: ITU metrics

214

Chapter 8 _ Classes of Service (CoS) _ Differentiation _ Prioritization _ Queue management _ Per CoS shaping and policing _ Penalties SLAs, Monitoring and Management _ Pre-deployment simulation and modeling _ Manufacturer-specific monitoring and management _ Real-time business views _ Call detail records _ Calls in progress _ Delay-to-dial-tone rates _ Gateway channel utilization and loading _ Real-time call monitoring _ Phone and multi-media device availability and monitoring _ Successful vs. failed call completion rates _ Poorly performing components _ Service level breaches and SLA compliance _ Real-time interface to Manager of Managers (such as HP OpenView) _ Summary and exception reporting _ Utilization trends over time _ Managed devices by company, department, and location _ Asset tracking _ Capacity planning _ Incoming and outgoing calls _ Loading by dial plan, routing rules and gateway. _ Bandwidth utilization _ Delay and delay variation (jitter) _ Packet loss _ Route patterns, utilization, and availability _ Customer Network Management (CNM) _ MSP provided tools _ Enterprise Tools _ Manufacturer provided tools _ Third-party tools

215

Chapter 8

Assessing Urgency and Prioritizing Migration _ No flash cut _ Assess urgency _ Prioritize migration order • Which office or region has outgrown, or is closest to outgrowing their existing system? •

Which region has bandwidth or IP network infrastructure good enough to carry voice, and other resources to accommodate the new telephony upgrades?

•

Which region has the most technically savvy support staff?

•

Which region was involved with early testing and/or staging and is, therefore, most familiar with the new system?

•

Which office or region is closest, because the earliest migrations are always learning experiences and should be expected to go slower and be more problematic?

Migration _ Systems Integrator vs. Do It Yourself _ Managed Service vs. Do It Yourself Planning & Assessment _ Inventory of existing capabilities _ Network infrastructure • Identify circuits used for wide area networking and Internet connectivity •

Inventory the network hardware—routers, switches, cabling. Don’t overlook elements like router operating system version. This is a good time to get your routed network all on a common foundation. _ Current voice services and equipment • Document any inbound and outbound call center environments •

List the details of hunt groups, call pickup groups and automatic all distribution groups. _ Current business data applications • What servers are in place today? What’s changing or evolving in parallel? Identify any linkages between your existing business applications and voice services. If you have a CRM system in place, this piece will be absolutely crucial. _ Security posture • Inventory your security environment early to ensure there are no surprises later on. _ Existing voice needs—The calling patterns today _Call Detail Records (CDRs)

216

Chapter 8 _ Network Readiness • Estimate the bandwidth needed to support a known number of phone lines •

Calculate the bandwidth required to support busy-hour call volumes

•

Determine how many paths might be needed across a wide area network (WAN)

•

Delay budget is important and includes all sources of fixed delay. Delay less than 150ms one-way is ideal; >300ms will impact voice QoE. Delay should be specified in the SLA.

•

Delay variation (aka jitter) is a bigger problem than delay. Delay is fixed and easier for the listener to adjust to. Delay variation is difficult to adjust for and causes anxiety and stress on the part of listener.

•

The human ear can adjust to packet loss up to about 10 percent. Packet loss >1.5 to 3 percent can have impact on voice quality. Packet loss impact can be reduced by shortened voice samples and fewer samples per RTP packet.

•

Network availability directly impacts QoE. No dial tone is a BIG problem and causes loss of confidence in the system. _ CODEC Selection _ Voice Coding & Quality _ Bandwidth Use _ Constant Bit Rate _ Variable Bit Rate _ Silence Suppression / Voice Activity Detection _ Gateways & Signaling _ Trunking Gateways _ Signaling _ Echo Cancellation _ Assessment Tools & Modeling Process _ Traffic collection from existing IP network _ Call traffic collection from existing telephony network _ Import of Call Detail Records (CDRs) from telephony network/bill _ Off-line simulation of VoP/VoIP network impact _Simulated VoP/VoIP traffic insertion into live network _ Real-time assessment of VoP/VoIP traffic impact _ Real-time tuning of VoP/VoIP parameters _ Manufacturer-specific parameters, metrics and analysis _ Call Server, IP-PBX or Switch Manufacturer-provided tool _ Simulate complex traffic patterns _ Modeling/sizing of softswitches, IP PBX and IP Centrex _ Modeling/sizing of gateways _Transfer pre-deployment assessment info to NMS 217

Chapter 8 _ Model different codecs _ Pulse Code Modulation (G.711/ PCM) _ Adaptive Differential Pulse Code Modulation (G.721/G.726/G.727 ADPCM) _ Linear Predictive (LP) codecs: G.729, G.729a or G.723.1, G.726 _ Other Proprietary Linear Predictive (LP) codecs _ Wideband codecs _ Different voice sample sizes _ Different packet fills _ Voice activity detection/silence suppression _ VoIP w/SIP _ VoIP w/H.323 _ Voice over Frame Relay (VoFR) _ Voice over ATM (VoATM) _ Voice over DSL (VoDSL) _ Modeling/sizing VoP gateway performance _ Performance through multiple gateways _ Delay _ Delay Variation / Jitter _ Service Level Agreement Classes of Service _ Multimedia VoIP/VoP calls with video _ Impact of VoIP/VoP on data and video performance _ Impact of data and video performance on VoP/VoIP _ Performance of multimedia sessions/applications _ Simulate combined multimedia nets (i.e. different divs or depts) _ Simulate multimedia MPLS VPN _ Simulate multimedia Metro or Wide Area Ethernet VPN _ Simulate multimedia VLANs _ Forecast Mean Opinion Score (MOS) using G.107 eModel _ Forecast Mean Opinion Score (MOS) using proprietary method _ Forecast Delay _ Forecast Delay Variation / Jitter _ Forecast VoP Service availability _ Model voice-only QoS/QoE _ Model non-voice multimedia QoS/QoE (i.e. video) _ Model combined multimedia QoS/QoE _ Model packet loss and impact _ Draft "Buy vs. Build" reports to support VoP/VoIP approach _ Cost/price analysis for design option _ Reports/tables/charts showing voice quality metrics vs costs _ Draft Return on Investment (RoI) Calculations _ Draft Total Cost of Ownership (TCO) Calculations _ Draft Cost/Minute or Cost/User breakdown _ Draft Cost/Minute or Cost/User analysis by SLA category _ SLA compliance and out-of-compliance reports 218

Chapter 8 Design and Pre-Deployment Testing _ The Team _ Current IP Network Technical _ Current IP Network Management _ Current Voice Technical _ Current Voice Management _ Temp/Contract Multi-Technical _ Current Multi-Technical (If they exist) _ New Multi-Technical _ IT/Network Management _ Consultant _ Manufacturer Professional Services _ Internet Service Provider _ Data Systems Integrator _ Voice Systems Integrator _ Telephony Service Provider _ Service/Feature Support _ Needed telephony features _ Voice Band Data (FAX, Modem) _ DTMF Support _ SLA Compliance Modeling and Prediction _ Predict service availability for voice/telephony services _ Predict MOS for voice/telephony services _ Predict service availability for data services _ Predict service availability for video/IPTV services _ Predict Time-to-Dial Tone (TDTT) for voice/telephony services _ Predict Call Set-Up Time for voice/telephony services _ Predict Call Tear-Down Time for voice/telephony services _ Predict delay for voice/telephony services _ Predict delay for data services _ Predict delay for video/IPTV services _ Predict delay variation for voice/telephony services _ Predict delay variation for data services _ Predict delay variation for video/IPTV services _ Predict packet loss for voice/telephony services _ Predict packet loss for data services _ Predict packet loss for video/IPTV services _ Distance sensitivity for delay and delay variation _ Predict high/low performance for mix traffic loads _ Predict Mean-Time-To-Repair _ Predict Mean-Time-Between-Failure _ Compare different SLA Scenarios _ Model Network-Only SLA Compliance _ Model Network+Access SLA Compliance 219

Chapter 8 _ Model End System-to-End System SLA Compliance _ Component Simulation _Simulate IPT Phone Performance • All codecs •

Compression

•

Encryption

•

Silence Suppression/Voice Activity Detection

•

Multi-Line

•

Multi-Media

•

Ethernet-10M,100M,1G,WiFi

_ Simulate VoIP SoftPhone Performance • All codecs •

Compression

•

Encryption

•

Silence Suppression/Voice Activity Detection

•

Multi-Line

•

Multi-Media

• Ethernet-10M,100M,1G,WiFi _ Simulate Analog Terminal Adapter Performance • All codecs •

Compression

•

Encryption

•

Silence Suppression/Voice Activity Detection

•

Multi-Line

•

Multi-Media

•

Ethernet-10M,100M,1G,WiFi

_ Simulate SoftSwitch/IP PBX Performance • Simultaneous Calls •

Busy Hour Call Attempts

•

Max number of stations

_ Simulate Gateway/BGC Performance • Total Simultaneous Calls •

Busy Hour Call Attempts

•

Security Filtering/Monitoring

_ Power & Power Back-Up 220

Chapter 8 _ QoS / QoE Forecasting _ Forecast IPT Mean Opinion Score (MOS) using G.107 E-Model _ Forecast Data QoS parameters _ Forecast Video QoS _ Simulate Differentiated Services QoS (DiffServ) _ Simulate 802.1p/q VLAN Prioritization _ Simulate Congestion Avoidance (RED, WRED, Flow-based WRED) _ Simulate QoS Packet Marking _ Simulate QoS Congestion Management (WFQ,CBWFQ, FIFO,LLQ, PQ) _ Simulate RTP Priority Mechanisms (IP & FR) _ Simulate RTCP Feedback Loops _ Simulate NetFlow application performance _ Simulate impact of access bandwidth differences _ Simulate impact of backbone bandwidth differences _ Validation of Design _ Design Tuning _ Test Bed Architecture and Proof of Concept Implementation and Migration _ Planning _ Plan Validation Visualization Role Play Naysayer Validation Non-Expert Validation Red Teams _ Implementation Plan _ Training Plan(s) _ Test/Acceptance Plan(s) _ Migration Plan(s) _ On-Going Operations Plan(s) _ Contingency Plan _ Business/System Continuity Plan _ Escalation Procedure(s) _ Security Issues _ Liaison with Security Department _ Firewalls & Proxy Server Issues _ IP Addresses & Port Numbers _ NAT, Internal and External IP Addresses _ Internal & External Phone number mapping _ Purchasing & Project Management _ Establish Site Profiles _ Rough-Order-of-Magnitude Pricing _ Management Review and Site Prioritization 221

Chapter 8 _ Initial Migration/Green Field Plan & Renegotiation of Favorable Contracts _ Joint Migration/Green Field Planning Effort _ Contracts & Delivery _ Surplus Equipment Management Effort _ Inventory & Fixed Asset Accounting & Disposition of Old Equipment _ Implementation of Changes to Business Operations _ Training _ Dealing with User Reluctance _ Standardization _ Impact Assessment _ Changes to Processes _ Staff Implications _ Review of Standards Status and Maturity _ Develop Private Net, IP VPN and Carrier/PTT Strategies _ Private Backbone Design/Upsize for Voice Traffic and Backbone Re-engineering _ Establish Common Standards and Test Plan _ Establish Private Net, IP VPN and Carrier/PTT Demo/Test Capabilities _ Training/Staging for Migration Teams _ Training for Support Teams _ Review of Planning and New Telephony Project Roll-Outs _ Legal _ Closing contracts for old telephony system _ New contracts _ Compliance with National Law & Telco/PTT Regulation _ Security & Privacy Issues _ Carrier/PTT IP Bypass & Local Tail Drop-Off _ PBX and Gateway Issues _ Contact Centers / Call Centers _ Special Considerations & Contingency Planning _ Phased Implementation & Call-taker Training

222

Chapter 8

Ongoing Operations _ Review the original plans, goals and objectives Are the original goals and objectives still valid? Have technologies changed significantly? Are standards more mature, and does it matter? Have laws or regulations in any of the areas in which we operate changed in any important ways? Have our operating procedures or business environment evolved in a way that impacts our new telephony system? _ Growth Management _ Fixed Assets and Equipment Budgets _ Network Bandwidth: Access and Backbone _ Shared Resources: Switches, Routers, Call Servers, IP PBXs, Gateways _ Numbers and Licenses _ Record Keeping & Documentation _ Fixed Asset Accounting _ Shipping / Warehousing / Tracking _ Repair / Refurbishment / Return to Service / End-of-Life Optimization _ Constant Testing and Monitoring _ Exception Reporting and Filtering _ “What If” Scenarios and Contingency Plans _ Trending and Capacity Planning _ Bandwidth _ Ports and Lines _ Phone Numbers and Numbering Plan _ Grooming _ VLAN Capacity _ VPN Capacity _ Infrastructure _ Hardware _ Repair _ Obsolescence / Refresh _ Software _ Maintenance _ Enhancements _ Upgrades _ Patches & Security Audit

223

Chapter 8 _ SLA Improvements _ Delay _ Call Set-up _ During Call _ Call Tear Down / Release _ Delay Variation _ Sample and Packet Loss _ Availability _ Telephony Service Availability _ Grade of Service (GoS) Measurement _ Busy Hour Call Attempts (BHCAs) _ Busy Hour Call Completions (BHCCs) _ Price and Cost Optimization _ Hardware Costs _ Service and Support Costs _ In-Sourcing vs. Out-Sourcing _ IP Contact Center Optimization Operations & Security _ Tracking of SLA parameters: packet loss _ Tracking of SLA parameters: delay _ Tracking of SLA parameters: delay variation/jitter _ Tracking of SLA parameters: availability _ Tracking of SLA parameters: user profiles _ Tracking of SLA parameters: voice quality metrics _ Tracking of SLA parameters: MTBF _ Tracking of SLA parameters: MTTR _ Tracking of SLA parameters: Bandwidth per call _ SLA Compliance Exception Reporting _ Tracking of protocols used _ Tracking of services and applications used _ Tracking of out-of-service time for circuits _ Tracking of out-of-service time for components _Availability tracking for TDM circuits _ Security events reporting: encryption and key management _ Security events reporting: tunnels and secure tunnels _ Security events reporting: user access info & violations _ Security events reporting: DoS attempts _ Security events reporting: firewall performance/blocked intrusions _ Security events reporting: SW release levels & patch history _ Security events reporting: Access Retries _ Security events reporting: Session Hijacking Attempts _ Security events reporting: Eavesdropping Attempts

224

Chapter 8 Administration _ Track/Audit service provider call detail records (CDR) _ Track/Audit bandwidth and resource usage by user and user profile _ Track/Audit minutes of use by user and user profile _ Track/Audit applications by user and user profile _ Track/Audit features and services used by user and user profile _ Reconcile Telco Carrier Billing _ Reconcile Managed Service Provider Billing _ Reconcile Service Provider Billing _ Automate SLA Compliance Penalties _ Report Calculated MOS by user and user profile _ Correlate MOS with SLA compliance _ Report costs per user and user profile Provisioning _ Remote configuration of hard phones _ Remote configuration of soft phones _ Remote configuration of call servers _ Remote configuration of IP PBXs _ Remote configuration of gateways and Session Border Controllers _ Implement Secure Shell (SSH) _ Supports multiple levels of service technician log-in _ Tracking of moves/adds/changes by service tech _ Automated fall-back to last known good configuration _ Automated bulk configuration using profiles and templates _ Pre-configuration of IP Phones and ATAs prior to shipping _ Tracking of user changes _ Support for Tripwire or similar anti-tampering functionality _ Other configuration stabilization and anti-tampering _ Support for secure FTP _ Support of Trivial File Transfer Protocol for config download _ Support of anonymous File Transfer Protocol for config download _ Support of Telnet for remote configuration

225

Chapter 8

Maintenance _ Track moves/adds/changes _ Track software levels, patches and upgrades _ Track firmware levels, patches and upgrades _ Track/Audit IP hard phones and ATAs _ Track/Audit IP soft phones _ Track/Audit IP PBXs _ Track/Audit IP gateways and Session Border Controllers (SBC) _ Track/Audit services and features used _ Testing of Services and Features _ Remote Testing of Services and Features _ Automated Testing of Services and Features _ Hardware preventative maintenance _ Spare inventory management _ Maintenance contract tracking _ Warranty tracking _ Service and Trouble Ticket Tracking _ Trouble Ticket Trends and Statistics _ Automated/Scripted Root Cause Analysis (RCA) _ Escalation Procedures

226