對非功能性屬性的測試（上）

原題：Testing for Non-functional Attributes

  ABSTRACT - System testing involves testing for non-functional as well as functional requirements, non-functional requirements being the ones relating to attributes like performance, usability, availability, etc. Though functionality testing is well planned in most projects, the same can not be said about non-functional testing. The test team needs to explicitly focus on non-functional testing during each phase of the testing lifecycle.

  This paper discusses as to how focus can be brought on the non-functional attributes during various phases of software test life cycle namely: test planning, test design, test execution and test reporting.

  摘要：

  系統測試不僅包含了對功能性需求的測試，還包含了對非功能性需求的測試，而非功能性需求會涉及到一些諸如性能、可用性、可行性等屬性。盡管在大多數項目中，能夠對功能性測試進行良好規劃，但是對非功能性測試卻很難這么做。因此，測試團隊需要在測試生命周期的每一個階段都非常重視非功能性測試。

  這篇論文研討了在測試生命周期的幾個不同階段——測試計劃、測試設計、測試執行、測試報告——對非功能性測試需要重點關心什么。

  KEY WORDS - Testing, test planning, requirements, performance, usability.

  關鍵詞： 測試、測試計劃、需求、性能、可用性

  1. Introduction

  1. 簡介

  Product quality means more than meeting the functional requirements of the product as stated by the customer. It is beneficial to the end user, the customer organization and the development organization to identify and improve the non functional requirements of the product like performance, usability, reusability, availability etc. The product design and implementation should ensure that the product meets the specified non-functional requirements. It is aspects like poor performance, usability etc which cause system re-design after delivery. In MIEL, as part of the product quality improvement initiative, projects are getting to focus explicitly on non-functional attributes during the requirements phase. These non-functional requirements which are explicitly captured in the requirements book, need to be verified/validated as well.

  Some non-functional requirements like evolvability can be verified through analysis methods like SAAM[2], while some others like usability and performance can be validated by testing. In this paper we are exploring the role of system testing in validating the non-functional requirements. We outline activities & strategies relating to non-functional attributes that can be done in various phases of software test life cycle: test planning, test design, test execution etc. Performance and usability testing in particular are covered in greater detail.

  In section 2, we discuss the software test lifecycle, and in section 3 the focus is on non-functional testing related activities on the different phases of the software test lifecycle. In section 4 we touch upon few key issues relating to non-functional testing. Sections 5 and 6 discuss testing for performance and usability respectively.

  產品質量意味著不止滿足客戶提出的功能性需求。這才有利于最終用戶、客戶組織以及開發組織確定并改進諸如性能、可用性、重用性、可行性等非功能性需求。產品的設計和實現應該務必使之能滿足特定的非功能性需求。很明顯，性能低、可用性低等情況往往會導致系統在提交之后需要重新設計。在MIEL中，作為產品質量實現的初始階段的一部分，項目在需求階段變的十分重視非功能性屬性。這些寫入需求說明書的非功能性需求同樣也需要被驗證和確定。

  一些像可演化性這樣的非功能性需求能夠通過分析模型來驗證，例如SAAM。而另一些像可用性以及性能這樣的需求則能夠用測試來確定。在這篇論文中，我們將探究系統測試在確定非功能性需求的過程中所扮演的角色。我們為非功能性屬性勾畫一系列的活動和策略，這些活動和策略在不同的軟件測試生命周期（測試計劃、測試設計、測試執行、測試報告）中可以被實施。其中，對性能和可用性測試會討論的更加詳細。

  在第二部分，我們將討論軟件測試生命周期，而在第三部分，重點是在軟件測試生命周期的不同階段與非功能性測試相關的活動。在第四部分，我們會接觸到一些非功能性測試的關鍵點，而在第五和第六部分，將分別討論對性能和可用性的測試。

  2. Software Test life Cycle

  2. 軟件測試生命周期

  Figure – 1 depicts the generic software development / maintenance life cycle followed in MIEL. Non-functional requirements need to be identified during the requirements phase[4]. The captured non-functional requirements are analyzed and some of them which are testable and are very crucial from customer perspective become part of the requirements book and others are input to the design as design goals.

  Figure-1 also identifies the different phases of software test life cycle. Please see table (1) for the different phases and their inputs, outputs and activities. Test case development and test software development can be parallel activities.

Non-functional requirements are fundamentally hard to verify and. Some requirements like usability and maintainability tend to be very subjective and therefore are not readily testable. In such cases it helps to identify objective attributes which quantify these aspects of the system. In other cases like stress and performance, simulation in the test lab is the challenge. This requires that extreme care should be taken in every phase of STLC to ensure that provisions are made to test for the identified non-functional attributes.

PHASE	INPUTS	PROCESS	OUTPUTS
Test Planning	ReqB, SPMP, SQAP, SCMP	Identify scope, strategy, effort, resources etc..	SSTP
Test RequirementSpecification	ReqB, SSTP, Operation profile	Identify test case, test software and setup requirements.Define categories of test to be performed – performance, conformance, stress etc..	TRS
Test Case Development	TRS, ReqB/ PSRS	Identify test cases and sequencing.	TCD, TSC
Test Software Development	TRS, ICD, TCD	Design, code and test the test software. Prepare user manuals	Test software code and documentation
Test Execution	TCD, TSC	Execute the test cases, document the results, defects and observations	Test Log Reports
Test Reporting	Test Logs	Analyze the test logs.	TPR, STSR
Table 1: Phases of software test life cycle

Category	Examples	Test Requirements	Test Procedure
Performance	· Traffic capacity · Transaction / data throughput · UI response time · Call setup time · Memory Utilization	· Instrumentation of the code to help in measurements (time) · Special tools to load the SUT and measure time, throughput.	· Identify the dependencies – configuration, external and system state · Identify the values the dependencies can take and decide on the different combinations on which testing will be done. · Execute tests and record measurements · Co-relate parameters and dependencies by plotting charts
Usability	· Installation · Routine Procedures · Documentation	· UI Subjects matching user profiles	· Identify critical procedures, sections of UI etc.. for which usability is of concern · Identify the subjects · Test with subjects – observe/videotape · Analyze
Reliability		Multiple rounds of system testing	· Collect the failure and execution time data from multiple rounds of testing. · Use a suitable software reliability model (e.g.: Goel and Okumoto,1979) to predict reliability
Security	Firewalls, Networks	Can not prove system is secure. Establish to sufficient degree of confidence that it is secure enough.	Use of tiger teams (professional hackers!!!)
Configuration	Hardware and software configurations Redundancy means more configurations		· Identify the combinations · Select important (more used or more susceptible) ones · Do a selected subset of test cases · Degradation and restoration testing (transition between configurations)
Volume	· No. of subscribers · Predefined load profile over time	Automation required
Stress	BHCA	Tool to generate the stress condition	Real loads can be used for Volume and stress testing – they are the best and worst kind of load to use for testing.
Recovery	Automatic restart Switchover		Simulate the condition to initiate recovery.Confirm resumption of processing without transaction compromise.
Table 2: Categories of non-functional testing

原文轉自：http://www.kjueaiud.com