典型測試錯誤（二）——計劃測試工作

　Theme Two: Planning the Testing Effort

　　主題二：計劃測試工作

　　I'll first discuss specific planning mistakes, then relate test planning to the role of testing.

　　我將首先討論特定的計劃錯誤，然后將測試計劃與測試作用關聯起來。

　　It's not unusual to see test plans biased toward functional testing. In functional testing, particular features are tested in isolation. In a word processor, all the options for printing would be applied, one after the other. Editing options would later get their own set of tests.

　　將測試計劃偏重于功能測試的情況的并不少見。在功能測試中，某個功能部件是孤立測試的。在字處理軟件中，所有打印選項都將一個接一個地應用。編輯選項在后面將得到它們自己的測試集。

　　But there are often interactions between features, and functional testing tends to miss them. For example, you might never notice that the sequence of operations open a document, edit the document, print the whole document, edit one page, print that page doesn't work. But customers surely will, because they don't use products functionally. They have a task orientation. To find the bugs that customers see - that are important to customers - you need to write tests that cross functional areas by mimicking typical user tasks. This type of testing is called scenario testing, task-based testing, or use-case testing.

　　但是，在各個功能部件中常常有交互作用，功能測試很容易遺漏它們。例如，你可能從未注意到一系列的操作：打開文檔、編輯文檔、打印整個文檔、編輯一頁、打印該頁不能工作。但是客戶一定會注意到，因為他們不會按功能使用產品。他們是面向任務的。如果要找到客戶看到的 bug——這些 bug 對于客戶來說是很重要的——你需要編寫模仿典型用戶任務的跨功能區的測試用例。這類測試稱為場景測試、基于任務的測試，或使用用例測試。

　　A bias toward functional testing also underemphasizes configuration testing. Configuration testing checks how the product works on different hardware and when combined with different third party software. There are typically many combinations that need to be tried, requiring expensive labs stocked with hardware and much time spent setting up tests, so configuration testing isn't cheap. But, it's worth it when you discover that your standard in-house platform which "entirely conforms to industry standards" actually behaves differently from most of the machines on the market.

　　偏重于功能測試也會低估配置測試的重要性。配置測試檢查產品在不同硬件上、以及在與不同的第三方軟件組合使用時如何工作。通常有不同的典型組合需要嘗試，需要有裝備了硬件的昂貴實驗室，并花費很多時間設置測試，所以配置測試成本不低。但是，當你發現你的“完全符合業界標準”的標準機構內部平臺實際上在市場上不同的機器上表現不同的時候，這樣做就值了。

　　Both configuration testing and scenario testing test global, cross-functional aspects of the product. Another type of testing that spans the product checks how it behaves under stress (a large number of transactions, very large transactions, a large number of simultaneous transactions). Putting stress and load testing off to the last minute is common, but it leaves you little time to do anything substantive when you discover your product doesn't scale up to more than 12 users.

　　配置測試和場景測試都測試產品的全面的、跨功能的方面。另一類測試是跨越產品以檢查在壓力(大量事務、很大的事務、大量并發事務)下的表現。將壓力測試和負載測試推遲到最后一刻才進行是一種常見的情況，但是這樣做的結果是，當你發現你的產品不能支持12個以上的用戶時，你已經沒有多少時間來采用實際的措施。

　　Two related mistakes are not testing the documentation and not testing installation procedures. Testing the documentation means checking that all the procedures and examples in the documentation work. Testing installation procedures is a good way to avoid making a bad first impression.

　　一個相關錯誤是不測試文檔，也不測試安裝過程。測試文檔意味著檢查文檔中所有過程和示例都能工作。測試安裝過程是避免給別人留下糟糕的第一印象的好方法。

　　How about avoiding testing altogether?

　　不做測試會怎么樣?

　　At a conference last year, I met (separately) two depressed testers who told me their management was of the opinion that the World Wide Web could reduce testing costs. "Look at [wildly suclearcase/" target="_blank" >ccessful internet company]. They distribute betas over the network and get their customers to do the testing for free!" The Windows 95 beta program is also cited in similar ways.

　　在去年的一個會議上，我(分別)遇到兩個沮喪的測試員，他們告訴我他們的管理是基于這樣一種意見：萬維網(World Wide Web)可以減少測試成本?！翱纯捶浅３晒Φ?STRONG>網絡公司”。他們在網絡上分發β版，讓客戶免費給他們做測試!”。Windows 95的β程序也是這樣的。

　　Beware of an overreliance on beta testing. Beta testing seems to give you test cases representative of customer use - because the test cases are customer use. Also, bugs reported by customers are by definition those important to customers. However, there are several problems:

　　要當心對β測試的過分依賴。因為測試用例是客戶使用的，所以β測試似乎是給了你客戶使用的代表用例。另外，客戶報告的錯誤也是對客戶重要的。但是，有幾個問題：

　　1. The customers probably aren't that representative. In the common high-tech marketing model, beta users, especially those of the "put it on your web site and they will download" sort, are the early adopters, those who like to tinker with new technologies. They are not the pragmatists, those who want to wait until the technology is proven and safe to adopt. The usage patterns of these two groups are different, as are the kinds of bugs they consider important. In particular, early adopters have a high tolerance for bugs with workarounds and for bugs that "just go away" when they reload the program. Pragmatists, who are much less tolerant, make up the large majority of the market.

　　客戶可能不是代表。在一個普通的高科技市場營銷模型中，β用戶，特別是那種“將產品放到網站上讓他們下載”的情況，是早期的采用者。他們喜歡擺弄新技術。他們不是實用主義者，不是那種愿意等到新技術被證明是安全可靠后才采用的人。這兩種類別的使用方式是不同的，就像他們認為 bug 的重要程度是不同的一樣。特別地，早期的采用者對于能夠用變通方法解決的 bug和重新加載程序就能消失的 bug有較強的容忍性。但容忍性較差的實用主義者占據了市場的大部分。

　　2. Even of those beta users who actually use the product, most will not use it seriously. They will give it the equivalent of a quick test drive, rather than taking the whole family for a two week vacation. As any car buyer knows, the test drive often leaves unpleasant features undiscovered.

　　即使是那些實際使用產品的β用戶，大多數也不會認真地使用。他們會給一個類似于試駕車的快速測試，而不是帶著整個家庭休假兩周。很多購買汽車的人都知道，試駕車經常會遺漏一些令人不愉快的特性。

　　3. Beta users - just like customers in general - don't report usability problems unless prompted. They simply silently decide they won't buy the final version.

　　β用戶象客戶一樣，除非特別要求，一般不會報告可用性錯誤。他們只是暗自決定不去購買最終產品。

　　4. Beta users - just like customers in general - often won't report a bug, especially if they're not sure what they did to cause it, or if they think it is obvious enough that someone else must have already reported it.

　　β用戶象客戶一樣，常常不會報告 bug ，尤其是當他們不能確定是什么操作導致了錯誤，或者是他們認為這個錯誤很明顯，其他人肯定已經報告了。

　　5. When beta users report a bug, the bug report is often unusable. It costs much more time and effort to handle a user bug report than one generated internally.

　　當β用戶報告錯誤時，錯誤報告常常無法使用。處理一個用戶的錯誤報告比一個內部產生的錯誤報告要花費多得多的時間和精力。

　　Beta programs can be useful, but they require careful planning and monitoring if they are to do more than give a warm fuzzy feeling that at least some customers have used the product before it's inflicted on all of them. See [Kaner93] for a brief description.

　　β程序可能是有用的，但是需要仔細的計劃和監督，否則它們在激怒所有β客戶之前，除了帶來一種模糊的、興奮的感覺，認為至少有一些客戶在使用產品之外，不會后其他收獲。參見[kaner93]以獲取一個簡要描述。

　　The one situation in which beta programs are unequivocally useful is in configuration testing. For any possible screwy configuration, you can find a beta user who has it. You can do much more configuration testing than would be possible in an in-house lab (or even perhaps an outsourced testing agency). Beta users won't do as thorough a job as a trained tester, but they'll catch gross errors of the "BackupBuster doesn't work on this brand of 'compatible' floppy tape drive" sort.

　　β測試有用的一種情況是配置測試。對于任何古怪的配置，你都可以找到一個使用此配置的β用戶。你可以做比機構內部實驗室(或者甚至是外包給測試機構)多的配置測試。β用戶不會象一個訓練有素的測試員一樣做完整的測試，但他們可以捕捉到大致錯誤，像“BackupBuster在這個品牌的兼容磁帶驅動器上不能工作”。

　　Beta programs are also useful for building word of mouth advertising, getting "first glance" reviews in magazines, supporting third-party vendors who will build their product on top of yours, and so on. Those are properly marketing activities, not testing.

　　β程序也有助于建立口頭的廣告，獲得雜志的“第一印象”評論，支持第三方供應商在你的產品上構建他們的產品等等。這些都是正常的市場營銷活動，不是測試。

　　Planning and replanning in support of the role of testing

　　計劃和重新計劃測試的支持作用

　　Each of the types of testing described above, including functional testing, reduces uncertainty about a particular aspect of the product. When done, you have confidence that some functional areas are less buggy, others more. The product either usually works on new configurations, or it doesn't.

　　上面所描述的包括功能測試在內的各種類型的測試，減少了產品某一方面的不確定性。在執行完畢后，你可以確信某些功能領域的錯誤較少了，其他的還比較多。產品通常將在新配置中起作用，或者是不起作用。

　　There's a natural tendency toward finishing one testing task before moving on to the next, but that may lead you to discover bad news too late. It's better to know something about all areas than everything about a few. When you've discovered where the problem areas lie, you can test them to greater depth as a way of helping the developers raise the quality by finding the important bugs.

　　有一種很自然的傾向，就是在進行到下一個測試任務之前先完成一個任務，但這可能導致你過晚地發現壞消息。對所有領域都了解一些比深入了解幾個領域更重要。如果你發現了問題在哪個地方，你可以更深入地測試它們，通過發現重要 bug來幫助開發人員提高質量。

　　Strictly, I've been over-simplistic in describing testing's role as reducing uncertainty. It would be better to say "risk-weighted uncertainty". Some areas in the product are riskier than others, perhaps because they're used by more customers or because failures in that area would be particularly severe. Riskier areas require more certainty. Failing to correctly identify risky areas is a common mistake, and it leads to misallocated testing effort. There are two sound approaches for identifying risky areas:

　　嚴格地說，我對將測試的作用描述為減少不確定性是太簡單了。更恰當的說法是“風險加權”的不確定性。產品中某些領域比其他領域更有風險，也許是因為它們由更多客戶使用或是因為那個領域的故障更嚴重。危險性高的區域需要更好的穩定性。不能正確地識別危險區域是一個常犯的錯誤，它導致測試工作的不恰當分配。

　　1. Ask everyone you can for their opinion. Gather data from developers, marketers, technical writers, customer support people, and whatever customer representatives you can find. See [Kaner96a] for a good description of this kind of collaborative test planning.

　　向每一個能夠找到的人征詢意見。從開發人員、市場人員、技術寫作人員、客戶支持人員和你能找到的每一個客戶代表那里收集意見。查看[Kaner96a]以獲得關于這種協同測試計劃的描述。

　　2. Use historical data. Analyzing bug reports from past products (especially those from customers, but also internal bug reports) helps tell you what areas to explore in this project.

　　使用歷史數據。分析以前產品的 bug 報告(特別是來自客戶的，但也要包含內部 bug 報告)可以幫助你辨別在這個項目中還需要探索哪些領域。

　　"So, winter's early this year. We're still going to invade Russia."

　　“今年冬天來得很早。但我們還是要入侵俄國?！?/P>

　　Good testers are systematic and organized, yet they are exposed to all the chaos and twists and turns and changes of plan typical of a software development project. In fact, the chaos is magnified by the time it gets to testers, because of their position at the end of the food chain and typically low status. One unfortunate reaction is sticking stubbornly to the test plan. Emotionally, this can be very satisfying: "They can flail around however they like, but I'm going to hunker down and do my job." The problem is that your job is not to write tests. It's to find the bugs that matter in the areas of greatest uncertainty and risk, and ignoring changes in the reality of the product and project can mean that your testing becomes irrelevant.

　　好的測試員是有計劃、有組織的，但他們受到計劃，特別是軟件開發項目計劃的各種混亂、各種意外轉折的影響，因為他們處于食物鏈的最后一環，而且通常地位比較低。一個不幸的反應是固執地堅持測試計劃。從感情上講，這會令人很滿意：“他們可以隨意胡亂擺弄，但我要坐下來做我的工作?！钡珕栴}是你的工作不是編寫測試。而是在最不確定和危險的領域發現 bug 。忽略產品和項目的實際變化可能意味著你的測試變得無關緊要。

　　That's not to say that testers should jump to readjust all their plans whenever there's a shift in the wind, but my experience is that more testers let their plans fossilize than overreact to project change.

　　這不是說測試員在有任何變化時都應該匆忙地重新調節他們的計劃，但我的經驗是很多的測試員都讓計劃僵化而不是對項目變化起過度的反應。

原文轉自：http://www.kjueaiud.com