Transcription
VA Enterprise Design Patterns:2. Enterprise Architecture2.2. End-to-End Application PerformanceManagement (APM)Office of Technology Strategies (TS)Architecture, Strategy, and Design (ASD)Office of Information and Technology (OI&T)Version 2.0Date Issued: December 2015
THIS PAGE INTENTIONALLY LEFT BLANK FOR PRINTING PURPOSESPage ii
REVISION HISTORYVersionDateOrganization1.09/25/14ASD TS2.012/4/15ASD TSNotesInitial version approved staffed for formalapproval by DCIO ASDUpdate accounting for lessons learned and realignment to current template for EnterpriseDesign PatternsREVISION HISTORY oleJoseph BrooksTim McGrailAPM Design Pattern LeadASD TS Design Pattern Final ReviewerPage iv
TABLE OF CONTENTS1INTRODUCTION . 11.11.22CURRENT CAPABILITIES AND LIMITATIONS. 22.12.22.32.43TRADITIONAL APM APPROACH .2CURRENT APM DEPLOYMENTS.3COMMON TECHNICAL CAPABILITIES .3CURRENT LIMITATIONS .5FUTURE CAPABILITIES . 53.13.23.34BUSINESS NEED .1APPROACH.2APM FOR MOBILE AND CLOUD SERVICES .5PROACTIVE PLANNING FOR APM .7ALIGNMENT TO TRM.8USE CASES . 8APPENDIX A.A.1A.2DOCUMENT SCOPE . 9SCOPE.9DOCUMENT DEVELOPMENT AND MAINTENANCE .9APPENDIX B.DEFINITIONS . 11APPENDIX C.ACRONYMS . 12APPENDIX D.REFERENCES, STANDARDS, AND POLICIES. 13APPENDIX E.ESS INTEGRATION WITH APM . 14ALIGNMENT TO ESS ARCHITECTURE CONSTRUCT .14ESS MONITORING APPROACH .15APPENDIX F.IDENTIFIED CURRENT PAIN POINTS IN APPLICATION PERFORMANCE . 17FIGURESFigure 1 – Traditional Monitoring Approach Providing Visibility to Specific Domains of a Typical BusinessTransaction. 3Figure 2 – APM Capabilities from the End User to Back-end Services and Databases . 4Figure 3 – End-to-end VA APM Capabilities and Transaction Visibility Conceptual Overview (Based onMaterials Provided by IBM) . 6Figure 4 – Process for APM with User Experience, Network, and Infrastructure Monitoring . 8Figure 5 – APM within the ESS Architecture Construct . 14Page v
TABLESTable 1 – APM Justification Themes for Business Owners . 1Table 2 – Definitions . 11Table 3 – Acronyms . 12Table 4 – References, Standards, and Policies . 13Page vi
1INTRODUCTIONInformation Technology (IT) projects within the Department of Veterans Affairs (VA) developedor acquired solutions in a stove-piped fashion, resulting in redundant functionality includingapplication performance management (APM). This duplication has resulted in an increasedtotal cost of ownership (TCO) and APM complexity. APM tools were not deployed end-to-end,hindering VA’s ability to evaluate application health consistently and to identify problemsproactively.This Enterprise Design Pattern provides guidance to projects in applying end-to-end APMcapabilities provided by Enterprise Shared Services (ESS). This document guides projects in theuse of standard APM capabilities provided by VA regional data centers. Additional coordinationwith OI&T Service Delivery and Engineering (SDE) Enterprise Operations (EO) on capacity andoperations planning is required prior to deployment.1.1Business NeedVA business owners derive a number of benefits from APM capabilities within the agency’s ITinfrastructure investments. APM provides business owners the following “justification themes,”according to APM Best Practices (see Appendix D):Table 1 – APM Justification Themes for Business OwnersJustification ThemeAvailability vs. PerformanceMonitoringResolving ApplicationIncidents and OutagesImproving ApplicationSoftware QualityPre-production ReadinessBenefits Enhances visibility into the behaviors of distributed systems and how tocorrelate and resolve various incidents Reduces the time to first alert for a performance incident Provides performance monitoring capability across transport protocols andJava/.NET platforms Enables efficient tracking and resolving performance issues Provides separate responses for availability and degradation incidents Allows more effective use of the monitoring tool infrastructure throughactive capacity reporting and planning Decreases overall time-to-market for new software systems Confirms accuracy and utility of load testing during development Improves production experience based on a consistent set of keyperformance indicators (KPI) Validates of low overhead of agent and transaction definitionsPage 1
and Deployment Supports definition of the monitoring dashboards and reportingManaging Service LevelAgreements (SLA) Enhances relationships with business owners Enables reliable transactions that are defined and focused Provides accurate and rapid performance and capacity forecasting Decreases time-to-market schedule Allows for optimal use of existing and proposed monitoring technology Helps evolve skill sets and competencies of technical staff Achieves proactive management by catching performance problems duringquality assurance (QA) and user acceptance testing (UAT) (DevOps) Enhances triage of performance problems Enhances overall software quality from the operations perspective Increases use of the monitoring environment Establishes comprehensive capacity management planning practices Establishes more capable triage technical practices Gives real-time view of business service performance Provides visibility into application component interactions and the end-userexperienceEnhancing the Value of theMonitoring ToolInvestmentProactive MonitoringTrending and AnalysisSingle-View of ServicePerformance (Dashboards)1.2 ApproachEnd-to-end APM is currently available at regional data centers to monitor all operationalsystems and services, including ESS (Appendix E). All new applications are required to integratethe APM capabilities provided by the VA data centers. The current approach applies tosolutions deployed at VA’s data centers and will accommodate VA-approved external cloudservice providers as the VA Cloud Strategy is deployed.22.1CURRENT CAPABILITIES AND LIMITATIONSTraditional APM ApproachThe following figure depicts VA’s monitoring approach focused on specific domains. Exampledomains include message queues (MQ), operating systems, and Java Virtual Machines (JVM).This approach does not provide the full visibility into an entire business transaction using alldomains, resulting in monitoring inefficiencies and a longer mean time to repair (MTTR).Page 2
ITObjectivesMODERNAPPROACHBusiness TransactionEND USER MONITORINGSilo’d domain visibility99.9%99.9%99.9%99.9%Figure 1 – Traditional Monitoring Approach Providing Visibility to Specific Domains of aTypical Business TransactionIndustry best practices recommend for end-to-end APM to leverage a top-down approachfocusing on the complete application stack. VA’s regional data centers (e.g., Austin InformationTechnology Center, Hines Information Technology Center, and Philadelphia InformationTechnology Center, etc.) offer the full spectrum of APM capabilities to monitor applicationhealth.2.2Current APM DeploymentsThe current end-to-end APM tools in VA data centers deliver a holistic view of all usertransactions, helping IT stakeholders understand the health, availability, service impact, andend-user experience of critical applications. APM enables projects to diagnose and resolveproblems proactively while optimizing the performance of mission critical services. APMsupports prioritization of incidents based on service impact and quickly pinpoints problemsacross disparate technology silos.2.3Common Technical CapabilitiesThe following figure describes APM products deployed by SDE Enterprise Operations (EO):Page 3
Figure 2 – APM Capabilities from the End User to Back-end Services and DatabasesThe following APM capabilities are available as enterprise infrastructure services:End-user Experience Monitoring – Ensures consistent end-user experience and high servicelevels meeting business objectives by monitoring all end-user transactions (including web-basedand non-web-based services) on a 24x7 basis with minimum overhead. APM measures end-usertransaction performance to ensure applications are delivering against service level agreements(SLA) using application-specific Key Performance Indicators (KPI).Application Behavior Analytics – Discovers anomalous application behavior automatically andproactively alerts IT operators of potential problems that could disrupt performance. Theinstrumentation tools provided by EO automatically mine the vast repository of rich datacreated by APM and, within hours of setup, can start identifying anomalous behavior incomponents, providing a view of potential issues between related components.Smart Triage – Reduces downtime and optimizes the performance of services by proactivelyidentifying, diagnosing and resolving performance problems before they impact end users. TheEO-provided APM tools map all transactions to the dependent infrastructure in real-time for asingle view of application health, business process flow, and the entire transaction path toPage 4
quickly triage issues, help eliminate problem resolution guesswork and accelerate mean time torepair.Rapid Root-cause Diagnosis – Improves IT productivity and controls costs by diagnosingproblems occurring within the application and infrastructure. End-user experience monitoringcapabilities integrate with behavior analytics and deep-dive problem diagnosis features tounderstand performance issues in context, pinpoint failures, and accelerate problem resolution.Business-centric Management – Assure high-value transactions receive the highest servicelevels by understanding problems in business context to identify critical transactions that maybe at risk, prioritize problem resolution efforts, dispatch the right resources, and fix theproblem affecting key functionality or end users. APM provides application health metricsunderstood by non-application experts and easily communicated to business users.2.4Current LimitationsThe current APM capabilities focus on the VA regional data centers and emphasize web-basedapplications. VA will also require the monitoring of cloud services and mobile applications in thefuture. PD and SDE will require upfront coordination prior to Milestone 1 to conduct capacityplanning and establishing KPIs prior to deployment. This necessitates a “DevOps” mindsetinvolving close collaboration between development and operations staff, especially as VA shiftsto a continuous integration and deployment paradigm.33.1FUTURE CAPABILITIESAPM for Mobile and Cloud ServicesThe future-state operational vision consists of end-to-end APM covering on-premise, cloud andhybrid environments, and support for DevOps practices for building, testing, and deployingapplications. The concept diagram below (source: IBM) depicts the types of services to beprovided by APM capabilities in various hosting environments.Page 5
Figure 3 – End-to-end VA APM Capabilities and Transaction Visibility Conceptual Overview(Based on Materials Provided by IBM)APM includes visibility into cloud environments, as discussed in detail in the Cloud ComputingEnterprise Design Patterns. APM integrates predictive analytics capabilities to enhanceproactive monitoring and trouble resolution. These capabilities support mobile applications andwill integrate with enterprise mobile analytics capabilities, as explained in the MobilityEnterprise Design Patterns. Specifically, end-user experience monitoring supports mobileanalytics, and APM achieves this through the following functions: Deploying a mobile performance agent on top of end-user monitoring capabilities,which may require adding a library and recompiling the code to perform APM for theapplicationAgent piggybacking on other user’s service calls through an application programminginterface (API)Generating crash analytics to create a snapshot of device crash statisticsDeploying APM for cloud services will include health endpoint monitoring. This is typically thecombination of two factors: the checks (if any) performed by the application or service inresponse to the request to the health verification endpoint, and analysis of the result by thetool or framework that is performing the health verification check. The response code indicatesPage 6
the status of the application and, optionally, any components or services it uses. Additional bestpractices include: Checking storage or a database for availability and response time Checking other resources or services located within the application, or locatedelsewhere but used by the application3.2 Proactive Planning for APMProactive planning for APM enables a DevOps paradigm that includes collaboration with bothdevelopers and operations staff. Projects will accomplish this through the following tasks: Establish KPIs with EO and conduct evaluations in a pre-production test environmentCoordinate infrastructure support and conduct operations support planning prior toMilestone 0 to establish monitoring KPIsDevelop a monitoring plan with known KPIs prior to Milestone 1Projects require a common set of KPIs to monitor, control and track relative to indicating poorperformance or equipment outage. Below is a representative list of KPIs: Message queue lengthTransaction or message throughput rateTransaction response timeDatabase query response timeEvent management statesGarbage collection behaviorFile I/O abnormalitiesPercentage of free storage space availablePercentage of network retransmissionsNetwork round trip timeNetwork connection timeSNMP connection failure (indicates complete equipment unavailability)Memory management patterns (e.g., JVM Heap)All new applications require load testing in pre-production environments. APM must beavailable in these environments to measure expected performance and identify potentialissues. Projects will work with EO to identify which of these KPIs needs to monitor during thetesting phase to mitigate pre-production performance risks. Appendix D contains technicalreferences on APM, and Appendix F includes pain points identified by EO to guide capacity andoperations planning.Page 7
3.3 Alignment to TRMAll APM products used in regional data centers are commercial-off-the-shelf (COTS) andcataloged in the TRM, and new APM products require TRM approval. APM aligns to thefollowing technology categories:Application Management, Monitoring, and NetworkPerformance Optimization.4USE CASESThe business process below shows a prescriptive flow for how end-to-end APM works withinthe VA enterprise for three scenarios:1. End user experience monitoring2. Network performance monitoring3. Back-end infrastructure monitoringMonitor & IDproblems fromthe end userexperienceEstablish SLAsfor ApplicationIntegrate Appwith APMcapabilityproviderProactivelydetect and logallperformanceproblemsMonitor & IDproblems atthe networklayerIsolateperformanceproblemsDiagnose rootcause ofperformanceproblemsReportperformanceproblems toapplicationownerMonitor & IDproblemswithin thebackendinfrastructureFigure 4 – Process for APM with User Experience, Network, and Infrastructure MonitoringThe basic flow of events between application owner and infrastructure actors (e.g., APMcapability provider) is as follows:1. Application owner establishes appropriate KPIs for the application in pre-production,including SLAs between service consumers and providers2. Application owner deploys application into the VA IT infrastructure productionenvironment and integrates with APM capability provider3. APM capability provider monitors all business transactions traversing the entire VA ITinfrastructure:a. Monitor and identify problems associated with the application layer (e.g., enduser experience)b. Monitor and identify problems associated with application delivery over thenetworkPage 8
4.5.6.7.c. Monitor and identify problems associated with the backend infrastructure (e.g.,application servers, web services, or databases)APM capability provider proactively detects and logs all performance problems in eachpart of the infrastructure (Parts 3a-c)APM capability provider isolates performance problems detected in Step 4APM capability provider diagnoses root cause of performance problems in Parts 3a-cAPM capability provider reports performance problems to application ownerAppendix A. DOCUMENT SCOPEA.1ScopeThe purpose of this document is to provide guidance on using end-to-end APM capabilities thatsupport Enterprise Shared Services (ESS). Specifically, this document guides projects to use thestandard set of APM capabilities provided by VA data centers and to coordinate with OI&TService Delivery and Engineering (SDE) Enterprise Operations (EO) early in the developmentlifecycle to ensure proactive performance monitoring. This applies to all new applicationsrequiring integration into VA’s enterprise IT. The guidance in this document applies to bothCOTS software (including open-source) acquisitions and applications developed by VA ProductDevelopment.This document focuses on APM capabilities provided by VA’s regional data centers. It provideshigh-level guidance for establishing application performance metrics (e.g., CPU usage, memorytrends, input, and output operations). The following content is out of scope for this documentbut addressed in related Enterprise Design Patterns: Mobile analytics (covered by Mobility Design Patterns) Log management for auditing and compliance (covered by Privacy and Security DesignPatterns) Network traffic monitoring (covered by Privacy and Security Design Patterns) Cloud service monitoring (covered by Cloud Computing Design Patterns) Vulnerability scanning and incident management processes (covered by IT ServiceManagement Design Patterns)A.2Document Development and MaintenanceThis Design Pattern was developed collaboratively with participation from VA’s Office ofInformation and Technology (OI&T), Product Development (PD), Office of Information Security(OIS), Architecture, Strategy and Design (ASD), Service Delivery and Engineering (SDE), andindustry. This document contains a revision history and revision approval log to track allPage 9
changes. Updates will be coordinated with the Government lead for this document, which willfacilitate stakeholder coordination and subsequent re-approval depending on the significanceof the change.Page 10
Appendix B. DEFINITIONSTable 2 – DefinitionsKey TermDefinitionEnterprise Shared Service A SOA service that is visible across the enterprise and(ESS)accessed by users across the enterprise, subject toappropriate security and privacy restrictions.ServiceA mechanism to enable access to one or more capabilities,where the access is provided using a prescribed interface andis exercised consistent with constraints and policies asspecified by the service description.Service OrientedArchitecture (SOA)A paradigm for organizing and utilizing distributedcapabilities that may be under the control of differentownership domains. It provides a uniform means to offer,discover, interact with, and use capabilities to producedesired effects consistent with measurable preconditionsand expectations.Service-Level(SLA)Agreement An agreement between two parties regarding a particularservice. They contain quantitative measurements that: Represent a desired and mutually agreed state of aservice Provide additional boundaries of a service scope (inaddition to the agreement itself) Describe agreed and guaranteed minimal serviceperformanceKey Performance Indicator Performancemetricstargetingservice(KPI)provider organization tactical and strategic objectives. Thesemetrics are used to measure: Efficiency and effectiveness of a service Service operation statusNot all metrics automatically become KPIs. KPIs must bebound to the organization or service goals and must drivecontinuous improvement and efficiency.Page 11
Appendix C. ACRONYMSTable 3 – AcronymsAcronymDescriptionAPMApplication Performance ManagementASDArchitecture, Strategy and DesignBPELBusiness Process Execution LanguageBAMBusiness Activity MonitoringBPMBusiness Process MonitoringCoECenter of ExcellenceCOTSCommercial Off-the-ShelfeMIEnterprise Messaging InfrastructureEOEnterprise OperationsETSPEnterprise Technology Strategic PlanIPTIntegrated Project TeamJVMJava Virtual MachineKPIKey Performance IndicatorMTTIMean Time to IdentifyMTTRMean Time to RepairPDProduct DevelopmentSDEService Delivery and EngineeringSLAService Level AgreementSNMPSimple Network Management ProtocolTCOTotal Cost of OwnershipPage 12
Appendix D. REFERENCES, STANDARDS, AND POLICIESTable 4 – References, Standards, and Purpose1VA OISVA 6500 HandbookDirective from OIS that establishes an informationsecurity program in VA, which applies to all applicationssubject to APM.2VA ASDVA Enterprise SOADesign PatternProvide a reference for the use of end-to-endapplication performance monitoring as part of theintegration with SOA support infrastructure services.These documents standardize and constrain thesolution architecture of all healthcare applications inthe VA.3VA ASDProvides the overarching strategy for developing,deploying, and managing ESS throughout the VA4VA ASD5VA ASDESS StrategyDocument andDirectiveVA EnterpriseTechnology StrategicPlan (ETSP)OIT InfrastructureArchitectureProvides long-term IT vision for systems managementcapabilities that include APMProvides a list of instrumentation/monitoring productsto be used (based on business/technical requirements)for the monitoring, proactive detection, triage anddiagnosis of performance problems in VA’s DataCenters.Additional technical references are as follows:1. APM Best Practices: Realizing Application Performance Management by Michael J.Sydor, 2010, CA Press, ISBN-10: 14302314162. px3. MSDN Application Architecture Guide v2, Chapter 17 Cross-cutting ee658105.aspxPage 13
Appendix E. ESS INTEGRATION WITH APMAlignment to ESS Architecture ConstructAPM capabilities monitor the performance of Enterprise Shared Services (ESS) using theapproved IT infrastructure hosted by VA’s data centers. APM is a platform capability thatconstitutes the service-oriented architecture (SOA) support infrastructure “backplane,” and itdoes not represent a specific business service, per the following ESS architecture layerconstruct:Figure 5 – APM within the ESS Architecture ConstructAPM monitors both front-end and back-end performance associated with common utilityservices shared across numerous applications meeting diverse business requirements. Per theESS Strategy document, new applications consuming ESS coordinate with the ESS Center ofExcellence (CoE) and follow applicable architecture guidelines provided by the CoE to ensureproper integration with ESS. APM is a crosscutting concern and referenced in platformarchitecture models. These models support service-specific architecture models for ESS inPage 14
alignment with business capabilities and drivers. APM also integrates with existing BusinessProcess Monitoring (BPM) and Business Activity Monitoring (BAM) functions in the EnterpriseMessaging Infrastructure (eMI), monitoring workflows and service orchestrations. APM alignsto the Open Group SOA Reference Architecture (Quality of Service Layer) by enabling thefollowing SOA management functionality: IT Systems Monitoring and Management: This category of capabilities providesmonitoring and management of IT infrastructure and systems. This includes the abilityto monitor and capture metrics and status of IT systems and infrastructure.Application and SOA Monitoring and Management: This category of capabilitiesprovides monitoring and management of software services and applications. Thisincludes the ability to capture metrics and to monitor and manage application andsolution status.Business Activity Monitoring and Management: This category of capabilities providesmonitoring and management of business activities and business processes. It providesthe ability to analyze this event information, both in real-time/near real-time, as well asstored (warehoused) events, and to review and assess business activities in the form ofevent information and determine responses or issues alerts/notifications.ESS Monitoring ApproachAPM unifies end-user experience and network performance monitoring through a singleappliance that provides a single source of truth on how network behavior affects the end-userexperience, making it faster and easier to identify, diagnose, and resolve transaction problemscaused by the network. APM provides application-aware infrastructure monitoring for any TCPbased application without desktop or server agents to deliver a consistent and common set ofresponse-time metrics, mitigate risks from planned changes and unexpected events, andresolve problems faster. By providing the TCP-level view of applications running over thenetwork and from tier-to-tier within the data center, it enables rapid troubleshooting ofnetwork and performance bottlenecks and provides insight into the duration, frequency,pervasiveness, and severity of problems. Automatic, intelligent baselines establish anunderstanding of normal performance, such that when deviations are detected, diagnostic datacan be gathered that helps further enable faster resolution of performance problems. All of thisinformation is accessible from a single, flexible APM dashboard for rapid troubleshooting andtriage.Page 15
The eMI contains products that combine information from WebSphere Service Registry andRepository (WSRR), observations and business process execution language (BPEL) businessprocess definitions. This allows users to: Reconcile services in WSRR with those monitored in target systems by the APM solutionTopology views show relationships between service operations and BPEL businessprocesses for impact analysisForwards status information to WSRR to allow selection of services based onperformance and other metricsPage 16
Appendix F. IDENTIFIED CURRENT PAIN POINTS IN APPLICATION PERFORMANCESDE EO has identified key pain points in performance for new VA applications. These relate tothe load and capacity testing capabilities, which enable measurement of these items prior toproduction. Mitigation of these pain points would remove 90% of the performance issues thatoperations encounter ac
APM includes visibility into cloud environments, as discussed in detail in the Cloud Computing Enterprise Design Patterns. APM integrates predictive analytics capabilities to enhance proactive monitoring and trouble resolution. These capabilities support mobile applications and will integr
