Blogs Web & API Performance Testing and Engineering with SmartQE Opensource Framework

Web & API Performance Testing and Engineering with SmartQE Opensource Framework

By Admin Oct 01, 2020

SmartQE, a performance testing and engineering opensource framework addresses the performance testing, tuning, post production monitoring, design & architecture evaluation of Web and API technology stack, using leading opensource performance testing and monitoring tools.

  • Do you know frequent downtime of application/product and performance issues damage the reputation of your brand?
  • Do you notice Application/product not cope with expected user load at normal and peak loads with desired response times?
  • Do you have to handle extended loads or sudden increases in load during normal business hours?
  • Are you sure about the performance of application is consistent and reliable on various browsers and operating systems?
  • Is your performance testing strategy revealing application behaviour at various conditions?
  • Are you sure about Reliability and consistence of application/system support future business growth?
  • Do you spend days’ time to setup performance test lab?
  • Organizations are challenges to achieving and maintaining their business’s mission critical applications at peak performance and scalability levels with optimal hardware and software resources.
  • Do you know the direct infrastructure update, migration will have risks and impact on overall output and fall behind the business requirements and delays in development?
SmartQE’s Key Features :

Agile Performance Testing - Performance test requirements are identified against the key functionality of an evolving application. This will add scope during subsequent sprints so that, over time, coverage will expand and any performance problems will identify early in the life cycle.

Tests are run automatically with continuous integration. Design and deliver performance test suites on the SmartQE and opensource tools and integrating with Jenkins and Docker container platform.

By running these tests regularly, confidence in application performance can be maintained throughout the development process. This reduces the application rework further down the line, when inevitable performance problems are discovered.

Agile Performance Engineering - Design and implement integrated Performance Testing into your development process that allowing development teams better, earlier visibility of potential performance problems identifying and fixing issues early in the life cycle.

Performance Testing
  • User scalability Test
  • Volume Test
  • Load & Scalability Test
  • Transaction Load Test
  • Reliability Test
  • Performance testing/ Response Time Test
  • Failover/Resilience Testing
  • Hardware Capacity Testing
  • Network Test executions
Architect & Design Evaluation
  • Define proactive architecture performance tuning parameters with respect to Hardware and Software
Tuning and Code Optimization
  • Application Tuning
  • Web, App and DB servers tuning
  • Load balancers & firewalls and network tuning
Post Production Monitoring Container code base performance testing
  • Docker code base performance validation at run time
Supported Domain & Technology
  • Web Applications / Web Services/ Client Server
  • CRM & Billing
  • Enterprise Solutions (Oracle Cloud App’s and SAP Fiori)
  • Data Warehousing / BI Technologies
  • Portals & Inventory Management
  • Middleware & Database
  • Oracle DB and IBM SOA
  • ECM and WMS
  • Telecom & IVR – Call Processing
  • Mobile
  • Salesforce and MuleSoft
  • ServiceNow
  • Workday
Supported Non-functional testing levels
  • Component level
  • API level
  • Application/Product Performance Testing
  • Web Service/Middleware Performance Testing
  • E2E Perf Testing
  • Interface and Standalone Performance Testing
  • Reliability Strategy management
  • Installation and Configuration Testing
  • Alert and High Availability Testing
  • Post production monitoring
Supported Non-functional testing types
  • Single User - It is conducted for assessing performance of the application when only single user is accessing the system
  • Light Load - It is conducted for validating the performance characteristics of the system when subjected to workload / load volume much lesser than what is anticipated during the production load.
  • Load - It is conducted for validating the performance characteristics of the system when subjected to workload / load volume what is anticipated during
  • Stress/Volume - It is conducted to make sure that the application can sustain more load than anticipated on production load
  • Recovery - It is conducted to make sure that the application is able to heal itself when load is decreased from failure point to stress point & then to normal load
  • Spike - It is conducted to find out the stability of the system when it is loaded in burst of very small time and releasing the load quickly to normal load.
  • Soak/Endurance/Reliability/Availability/Stability - It is conducted to find if system is capable of handling expected load without any deterioration of response time / throughput when run for a longer time.
  • Resilience - It is conducted to make sure that the system is capable of coming back to initial state (from stressful state to load level state) when it is stressed for short duration.
  • Failure - It is conducted to find out the capability of the application in terms of load. The load (concurrent users or volume) is increased until the application crashes.
  • Network Testing - Network performance test primarily tests the uplink and downlink speed of a network. It defines how quick and responsive a network is to user/data communication under different bandwidth.
Test phases and activities
Scoping & Requirement gathering and Planning
  • Understand Application technical and business overview
  • Collect business process flows/KBTs and test artifacts
  • Establish system access to Performance testing and engineering team
  • Execute KBTs in QA environment to understand business process flows and requirements
  • Capture SLAs for key business transactions & system resources
  • Validate work load profile for each KBT (No of Users & Transactions per hour per, day and peak, non-peak usage patterns and anticipated user and data volume growth)
  • Categorize test scripts/KBTs for various test types (Load/Stress/Endurance) including workload profile for each test type
  • Identify inbound and outbound integrations with Web and API
  • Perform feasibility analysis and determine suitable performance testing tools
  • Identify handoffs with development and other partner teams
  • Create Performance test strategy
  • Create test data strategy
  • Automate synthetic data creation scripts using automation tool
  • Design test scripts for identified KBTs
  • Finalize SLAs, Metrics and Governance cadence
  • Install and configure Prolifics test accelerators and PT framework
  • Design test scenarios in the performance test tool for various test types Load, Stress and Endurance with different workload profiles
Execution & Results analysis
  • Execute various test types (Component, String and System Integration testing) with required load profiles
  • Review and finalize regression suite
  • Setup load generation for test execution with different test types and load profiles
  • Capture test results and compare with baseline metrics and SLAs
  • Develop dashboards with test execution metrics
  • Correlate test results with back end trace and utilization metrics to identify performance bottleneck in Web and API apps
  • Validate Performance Test Results against the business SLAs
  • Log Performance bottlenecks and execute retesting
  • Closely work with BD, N/W and support teams to tune app the queries and create required indexes
  • Create lessons learned document for each release with bottlenecks, root cause analysis & proposed solutions
  • Create and publish Performance QA risk certification and notification document and sign off release regression testing
  • Bring AI powered analytics to probability of failure of systems and functions by learning about defect history & patterns. Auto-learn & Predict
Performance Engineering & Monitoring
  • Capture response times in the central access point for analyzing performance problems in the Web and API apps. This transaction is a crucial transaction as this helps to calculate the volume of transactions done in the whole testing
  • Determines how various buffers are performing in terms of percentage utilization and color-coded to help identify potential issues quickly
  • Database performance analysis monitor determines the mix of reads to writes, sequential to direct operations, physical to logical disk accesses, number of changes/updates/inserts, quality and size of key database buffers, and much more
  • OS level monitor to capture CPU status in terms of the workload queued over the last 1, 5, and 15 minutes. Also capture real-time and historical data related to memory swapping/paging, low-level disk access and disk queue details, and similarly low-level network access details
Performance Test Regression Library Maintenance
  • Capture before and after release changes
  • Collect KBTs to be added, deleted and modified in the performance regression library
  • Prioritize new test KBTs and conduct performance feasibility study
  • Customize and develop performance test scripts for new KBTs
  • Include newly developed performance test scripts in regression library and socialize release teams
  • Monthly connect with business to update load profile
Performance Testing Metrics
  • Throughput & Hits Per Sec
    • How many processes or units of work are completed in a second/hour/day?
    • What are peak vs average throughput rates & Total Hits Per Sec?
    • Webpage Diagnostics
  • Response times
    • How long do users wait for their activities, such as refresh task list, complete a task, click on next page?
    • Measure/Gather Individual Response Times for each API
  • Number of concurrent users
    • How many simulated users are used in the testing?
    • Work Load statistics
    • Histogram Metrics
  • Resource utilization at all Layers
    • CPU utilization
    • Memory utilization
    • I/O and network utilization
    • Single Point of Failure, Alert Mechanism & High Availability
  • Mobile Performance Test Metrics
    • Battery Time (%)
    • Network received (KB/s) and Network Transmitted (KB/s)
    • Used Physical Memory (MB)
    • CPU (%)
App Post Production Monitoring Report metrics
  • Transaction Analysis
  • Location Analysis
  • Response Time Over Time
  • Availability Over Time
  • Error Summary
  • Breakdown over Time
  • Breakdown Summary
  • Min. / Max. Response Time
  • Response Time by Percentile
  • Complete user information
  • Snapshot on error
  • Measuring User Activity
  • Content error detection
  • Complete real-user session capture
  • Business and information event capture
Business Value Addons
  • Your critical applications reliability and consistent performance is the key success to business
  • SmartQE evaluates your application and system stability under a sustained workload. The simulated workload will be maintained for a period of time (mention by the Business & Development teams).
  • SmartQE validates critical, high revenue generated and customer facing application features behaviour.
  • SmartQE performs early detection of bottlenecks.
  • SmartQE reduces development costs & time to market
  • SmartQE reduces infrastructure costs through on-demand test environments provisioning
  • SmartQE identifies and focus on the weakest component amongst critical components for performance improvements
  • SmartQE performs prediction of the breakpoints based on the marketing projections in terms of growth and tentative milestone months
  • SmartQE performs early prediction of transactions and/or servers potentially affecting performance due to increased resource utilization
  • Enabled right sizing of servers and load balancing in clustering for the expected loads.
  • Estimation of transactions per sec (TPS) capacity for Service delivery platform with reference to the impact of additional workflows on top of the existing transactions
  • SmartQE provides accurate performance test scenarios and monitoring during test execution which will help to identify the pain areas
  • Zeroed down the critical transactions which need to be monitored and/or enhanced by analyzing production response times and transactional data on various performance parameters
  • Kubernetes Docker and container platform, will help to reduce deployment and configuration time. Optimize hardware and software resources utilization. SmartQE helps you to automate test lab setup with required hardware and software’s to support parallel executions against different environments. Kubernetes dynamic auto scaling option can extend hardware on demand, once test execution complete it destroys containers to free hardware resources
  • SmartQE recommends tuning methods within the application flow for enhancements to help around 50% improvement in response times post release.
Share this post