典型測試錯誤（四）——工作中的測試員

主題四：工作中的測試員

　　When testing, you must decide how to exercise the program, then do it. The doing is ever so much more interesting than the deciding. A tester's itch to start breaking the program is as strong as a programmer's itch to start writing code - and it has the same effect: design work is skimped, and quality suffers. Paying more attention to running tests than to designing them is a classic mistake. A tester who is not systematic, who does not spend time laying out the possibilities in advance, will overlook special cases. They may be the same subtle ones that the programmers overlooked.

　　在測試的時候，必須決定如何執行程序，然后完成它們。完成它們比決定它們要有趣的多。測試員渴望的是開始破壞程序，程序員渴望的是開始寫代碼——這導致相同結果：設計工作被忽略了，產品質量受到損害。將更多的注意力集中于運行測試而不是設計它們是一個典型錯誤。

　　Concentration on execution also results in unreviewed test designs. Just like programmers, testers can benefit from a second pair of eyes. Reviews of test designs needn't be as elaborate as product design reviews, but a short check of the testing approach and the resulting tests can find significant omissions at low cost.

　　集中于執行測試也導致未經審核的測試設計。就像程序員一樣，測試員也得益于第二雙眼睛的檢查。測試設計的審核不必像產品審核那樣嚴格，但是對測試方法和結果測試的快速檢查可以低成本地找到重要的疏忽。

　　What is a test design?

　　什么是測試設計?

　　A test design should contain a description of the setup (including machine configuration for a configuration test), inputs given to the product, and a description of expected results. One common mistake is being too specific about test inputs and procedures.

　　測試設計應當包含設置描述(包括配置測試的機器配置)，對產品的輸入和預期結果的描述。一個常見錯誤是對測試輸入和過程過于注重細節。

　　Let's assume manual test implementation for the moment. A related argument for automated tests will be discussed in the next section. Suppose you're testing a banking application. Here are two possible test designs:

　　讓我們先假設一個手工測試實施。相關的自動化測試將在下一節討論。假設你在測試銀行應用程序。這里有兩個可能的測試設計：

　　Design 1

　　設計1

　　Setup: initialize the balance in aclearcase/" target="_blank" >ccount 12 with $100.

　　設置：將帳戶12的余額初始化為$100。

　　Procedure:

　　過程：

　　Start the program.

　　Type 12 in the Account window.

　　Press OK.

　　Click on the 'Withdraw' toolbar button.

　　In the withdraw popup dialog, click on the 'all' button.

　　Press OK.

　　Expect to see a confirmation popup that says "You are about to withdraw all the money from this account. Continue?"

　　Press OK.

　　Expect to see a 0 balance in the account window.

　　Separately query the database to check that the zero balance has been posted.

　　Exit the program with File->Exit.

　　啟動程序。

　　在帳戶窗口中輸入12。

　　按“確定”按鈕。

　　點擊“取款”工具條按鈕。

　　在彈出的取款對話框中，點擊“所有”按鈕。

　　按“確定”按鈕。

　　預期會看到一個確認消息：“您將從此帳戶中取出所有的錢，是否繼續?”

　　按“確定”按鈕。

　　在帳戶窗口中預期會看到余額為0。

　　單獨查詢數據庫，檢查余額為0。

　　通過“文件->退出” 退出程序。

　　Design 2

　　設計2

　　Setup: initialize the balance with a positive value.

　　設置：將帳戶余額初始化為一個正值。

　　Procedure:

　　過程：

　　Start the program on that account.

　　Withdraw all the money from the account using the 'all' button.

　　It's an error if the transaction happens without a confirmation popup.

　　Immediately thereafter:

　　- Expect a $0 balance to be displayed.

　　- Independently query the database to check that the zero balance has been posted.

　　啟動該帳戶的程序。

　　用“所有”按鈕從帳戶中取出所有的錢。

　　如果在事務發生時沒有彈出確認消息，則是一個錯誤。

　　其后立即：

　　- 預期余額會顯示$0。

　　- 單獨查詢數據庫，檢查余額為0。

　　The first design style has these advantages:

　　第一種設計風格有以下優點：

　　· The test will always be run the same way. You are more likely to be able to reproduce the bug. So will the programmer.

　　· 測試總是以相同方式運行。重現錯誤的可能性更大。程序員也一樣。

　　· It details all the important expected results to check. Imprecise expected results make failures harder to notice. For example, a tester using the second style would find it easier to overlook a spelling error in the confirmation popup, or even that it was the wrong popup.

　　· 它將所有要檢查的預期結果的細節都描述出來。不精確的預期結果使得錯誤更難注意到。例如，使用第二種風格的測試員將會發現更容易忽略確認對話框中的錯誤拼寫，甚至是錯誤的對話框。

　　· Unlike the second style, you always know exactly what you've tested. In the second style, you couldn't be sure that you'd ever gotten to the Withdraw dialog via the toolbar. Maybe the menu was always used. Maybe the toolbar button doesn't work at all!

　　· 不像第二種測試風格，你總是能明確地知道你在測試什么。在第二種風格中，你不能確定可以通過工具條得到“取款”對話框。也許總是使用菜單。也許工具條根本不起作用!

　　· By spelling out all inputs, the first style prevents testers from carelessly overusing simple values. For example, a tester might always test accounts with $100, rather than using a variety of small and large balances. (Either style should include explicit tests for boundary and special values.)

　　· 通過寫出所有的輸入，第一種風格防止程序員無意間過度使用簡單的值。例如，一個測試員可能總是用$100測試帳戶，而不是使用一些小的和大的余額的組合。(這兩種風格都應顯式地包含邊界值和特殊值測試。)

　　However, there are also some disadvantages:

　　但是，也有一些缺點：

　　· The first style is more expensive to create.

　　· 創建第一種風格的測試成本較高。

　　· The inevitable minor changes to the user interface will break it, so it's more expensive to maintain.

　　· 對用戶界面的一些不可避免的更改將中斷它，因此維護成本也就更高。

　　· Because each run of the test is exactly the same, there's no chance that a variation in procedure will stumble across a bug.

　　· 因為每一輪測試都完全相同，所以也就沒有機會因為過程不同而偶然發現 bug 。

　　· It's hard for testers to follow a procedure exactly. When one makes a mistake - pushes the wrong button, for example - will she really start over?

　　· 測試員難于遵循測試過程。如果一個人出現錯誤——比如說按錯按鈕——她需要重新開始嗎?

　　On balance, I believe the negatives often outweigh the positives, provided there is a separate testing task to check that all the menu items and toolbar buttons are hooked up. (Not only is a separate task more efficient, it's less error-prone. You're less likely to accidentally omit some buttons.)

　　如果能有一個獨立的測試任務來檢查所有的菜單項和工具條按鈕都連接了代碼(一個單獨的測試不但更有效，而且不易出錯。你不大會偶然地忽略掉一些按鈕。)，那么權衡利弊，我相信第一種設計的負面影響超過正面影響。

　　I do not mean to suggest that test cases should not be rigorous, only that they should be no more rigorous than is justified, and that we testers sometimes error on the side of uneconomical detail.

　　我不是認為測試用例不應當嚴格，只是說它們過分嚴格，而且我們測試員有時在不經濟的細節中犯錯誤。

　　Detail in the expected results is less problematic than in the test procedure, but too much detail can focus the tester's attention too much on checking against the script he's following. That might encourage another classic mistake: not noticing and exploring "irrelevant" oddities. Good testers are masters at noticing "something funny" and acting on it. Perhaps there's a brief flicker in some toolbar button which, when investigated, reveals a crash. Perhaps an operation takes an oddly long time, which suggests to the attentive tester that increasing the size of an "irrelevant" dataset might cause the program to slow to a crawl. Good testing is a combination of following a script and using it as a jumping-off point for an exploration of the product.

　　詳細的預期結果比詳細的測試過程問題要少，但是過多的細節可能是測試員的注意力過多集中于檢查他所依照的腳本。這可能也導致另一個典型錯誤：不能注意和探索“不相關的”奇怪現象。好的測試員善于注意到“有趣的東西”并對其進行操作?？赡茉诠ぞ邨l的一個短暫的閃動，經過調查后，揭示了一個失效錯誤。也許一個操作任務奇怪地花費了很長時間，可能使專注的程序員感到增加“不相關”的數據集合的大小可能使程序慢如蝸牛。好的測試是既遵循腳本，又能將它作為探索產品的出發點。

　　An important special case of overlooking bugs is checking that the product does what it's supposed to do, but not that it doesn't do what it isn't supposed to do. As an example, suppose you have a program that updates a health care service's database of family records. A test adds a second child to Dawn Marick's record. Almost all testers would check that, after the update, Dawn now has two children. Some testers - those who are clever, experienced, or subject matter experts - would check that Dawn Marick's spouse, Brian Marick, also now has two children. Relatively few testers would check that no one else in the database has had a child added. They would miss a bug where the programmer over-generalized and assumed that all "family information" updates should be applied both to a patient and to all members of her family, giving Paul Marick (aged 2) a child.

　　一個重要的忽略 bug的特例情況是檢查產品完成預期操作，但不檢查它是否沒有完成不應該完成的操作。舉個例子，假設你有一個更新醫療機構的家庭記錄數據庫的程序。一個測試是在Dawn Marick的記錄中添加第二個小孩。幾乎所有的測試員都將在更新之后檢查Dawn Marick現在有兩個小孩了。部分測試員——那些聰明的、有經驗的專家——將會檢查Dawn Marick的配偶——Brian Marick，現在也有兩個小孩了。相對較少的測試員將檢查數據庫中沒有其他人添加了小孩。如果程序員將規則過分擴展，認為應當對所有的既是病人又是她的家庭成員的人都更新 “家庭信息”，給了Paul Marick(2歲)一個小孩，則這個 bug 就被忽略了。

　　Ideally, every test should check that all data that should be modified has been modified and that all other data has been unchanged. With forethought, that can be built into automated tests. Complete checking may be impractical for manual tests, but occasional quick scans for data that might be corrupted can be valuable.

　　理想情況中，每個測試都應檢查需要修改的數據都被修改了，其他數據都沒有。在經過仔細考慮后，可以將這個過程構建到自動化測試中。完全檢查可能對于手工測試來說不切合實際的，但是偶爾地快速檢查數據是否破壞可能是很有價值的。

　　Testing should not be isolated work

　　測試不應當是孤立的工作

　　Here's another version of the test we've been discussing:

　　這里是我們討論過的另一個版本：

　　Design 3

　　設計3

　　Withdraw all with confirmation and normal check for 0.

　　取出所有錢，需要確認，并檢查余額為0。

　　That means the same thing as Design 2 - but only to the original author. Test suites that are understandable only by their owners are ubiquitous. They cause many problems when their owners leave the company; sometimes many month's worth of work has to be thrown out.

　　除了最初的作者，這與設計2是相同的。測試套件只有它們的作者才能理解是常見情況。當它們的擁有者離開公司后，會帶來許多問題;有時候很多個月的工作就白費了。

　　I should note that designs as detailed as Designs 1 or 2 often suffer a similar problem. Although they can be run by anyone, not everyone can update them when the product's interface changes. Because the tests do not list their purposes explicitly, updates can easily make them test a little less than they used to. (Consider, for example, a suite of tests in the Design 1 style: how hard will it be to make sure that all the user interface controls are touched in the revised tests? Will the tester even know that's a goal of the suite?) Over time, this leads to what I call "test suite decay," in which a suite full of tests runs but no longer tests much of anything at all.

　　我需要說明的是像設計1和2那樣詳細的設計也存在同樣的問題。雖然他們可能由任何人運行，但不是每個人都能在產品界面變化后更新它們。因為測試不會顯式地列出它們的目的，更新它們可能很容易使得比以前測試的少一點點。(例如，考慮一下，設計1風格中的測試套件：要確保所有用戶界面控件在更改后的測試中被涉及是一件多么困難的事情?)長期以來，這導致了我稱為“測試套件變質”的問題，完整的測試套件仍舊在運行，但什么也測試不了。

　　Another classic mistake involves the boundary between the tester and programmer. Some products are mostly user interface; everything they do is visible on the screen. Other products are mostly internals; the user interface is a "thin pipe" that shows little of what happens inside. The problem is that testing has to use that thin pipe to discover failures. What if complicated internal processing produces only a "yes or no" answer? Any given test case could trigger many internal faults that, through sheer bad luck, don't produce the wrong answer.

　　另一個典型錯誤是測試員與程序員的邊界。某些產品主要是用戶界面;他們做的所有操作在屏幕上都是可見的。其他產品主要是內部的;用戶界面是一個“細管道”，很少顯示內部發生什么。問題是測試必須使用那個細管道來發現錯誤。如果一個復雜的內部處理產生的只是“是或否”的答案，結果會怎么樣呢?任何給定的測試用例都能觸發很多內部錯誤，僅僅通過不壞的運氣，才不會產生錯誤的答案。

　　In such situations, testers sometimes rely solely on programmer ("unit") testing. In cases where that's not enough, testing only through the user-visible interface is a mistake. It is far better to get the programmers to add "testability hooks" or "testpoints" that reveal selected internal state. In essence, they convert a product like this:

　　在這樣的情況中，有時候測試員單獨依賴于程序員(“單元”) 測試。在這不夠充足的情況下，僅從用戶可見的界面測試是一個錯誤。如果使程序員加上“可測試性鉤子”或“測試點”以揭示所選擇的內部狀態的話，會好得多。本質上，他們將一個產品：

　　 

　　

　　

　　to one like this:

　　轉化為：

　　

　　 

　　

　　It is often difficult to convince programmers to add test support code to the product. (Actual quote: "I don't want to clutter up my code with testing crud.") Persevere, start modestly, and take advantage of these facts:

　　說服程序員向產品中添加測試支持代碼常常是很困難的(一個真實引語：“我不想讓測試代碼弄亂我的程序?！?堅持下去，適時開始，并利用以下事實：

　　1. The test support code is often a simple extension of the debugging support code programmers write anyway.

　　測試支持代碼常常只是程序員隨便編寫的調試支持程序的簡單延伸。

　　2. A small amount of test support code often goes a long way.

　　少量的測試支持代碼常常就會帶來很大幫助。

　　A common objection to this approach is that the test support code must be compiled out of the final product (to avoid slowing it down). If so, tests that use the testing interface "aren't testing what we ship". It is true that some of the tests won't run on the final version, so you may miss bugs. But, without testability code, you'll miss bugs that don't reveal themselves through the user interface. It's a risk tradeoff, and I believe that adding test support code usually wins. See [Marick95], chapter 13, for more details.

　　對這種方法的普遍的反對意見是測試支持代碼必須編譯在最終產品之外(以避免顯示)。如果是這樣的，測試員使用的測試界面“不是我們交付的產品”。誠然，某些測試不會運行在最終版本中，所以可能會漏掉一些 bug 。但是，沒有可測試的代碼，你會漏掉一些通過用戶界面無法揭示的 bug 。這是一個風險的權衡，我相信添加測試代碼通常會占上風。參見[Marick95]的第13章以獲取更多詳細內容。

　　In one case, there's an alternative to having the programmer add code to the product: have a tool do it. Commercial tools like Purify, Boundschecker, and Sentinel automatically add code that checks for certain classes of failures (such as memory leaks). They provide a narrow, specialized testing interface. For marketing reasons, these tools are sold as programmer debugging tools, but they're equally test support tools, and I'm amazed that testing groups don't use them as a matter of course.

　　有一種情況是，有一個方案替代程序遠向產品添加代碼：用工具來完成。一些商用工具如Purify、Boundschecker和Sentinel可以自動添加代碼以檢查某種類型的錯誤(比如內存泄露)。它們提供一個狹小的、專用的測試界面。因為市場營銷的原因，這些工具是作為程序員調試工具出售的，但它們等同于測試支持工具，測試小組沒有把它們當成常規工具來使用，讓我覺得很吃驚。

　　Testability problems are exacerbated in distributed systems like conventional client/server systems, multi-tiered client/server systems, Java applets that provide smart front-ends to web sites, and so forth. Too often, tests of such systems amount to shallow tests of the user interface component because that's the only component that the tester can easily control.

　　測試問題在分布式系統中，比如傳統的客戶/服務器系統、多層的客戶/服務器系統、向站點提供靈巧的前端應用的Java小程序等，可測試性問題更為嚴重。常常地，測試這類系統等同于用戶界面部件的淺顯測試，因為它們是測試員能夠容易控制的唯一部件。

　　Finding failures is only the start

　　發現錯誤僅僅是開始

　　It's not enough to find a failure; you must also report it. Unfortunately, poor bug reporting is a classic mistake. Tester bug reports suffer from five major problems:

　　發現錯誤是不夠的，還必須報告它。不幸的是，低劣的 bug 報告是一個典型錯誤。測試員的錯誤報告存在五個主要問題：

　　1. They do not describe how to reproduce the bug. Either no procedure is given, or the given procedure doesn't work. Either case will likely get the bug report shelved.

　　他們沒有描述如何重現 bug 。要么沒有描述過程，要么描述的過程不正確。這兩種情況都會使錯誤報告被擱置。

　　2. They don't explain what went wrong. At what point in the procedure does the bug occur? What should happen there? What actually happened?

　　他們沒有解釋出了什么問題。在什么地方出現了 bug ?將會發生什么?實際上又發生了什么?

　　3. They are not persuasive about the priority of the bug. Your job is to have the seriousness of the bug accurately assessed. There's a natural tendency for programmers and managers to rate bugs as less serious than they are. If you believe a bug is serious, explain why a customer would view it the way you do. If you found the bug with an odd case, take the time to reproduce it with a more obviously common or compelling case.

　　4. 關于 bug 的級別沒有說服力。你的工作是評估 bug 的嚴重性。對于程序員和經理有一種很自然的傾向：評估的嚴重性比實際的嚴重性低。如果你確信一個 bug 是嚴重的，要解釋一下為什么客戶要以你的方式看待這個問題。如果你發現一個奇怪的錯誤，花一些時間以更普通、更令人信服的方式重現它。

　　5. They do not help the programmer in debugging. This is a simple cost/benefit tradeoff. A small amount of time spent simplifying the procedure for reproducing the bug or exploring the various ways it could occur may save a great deal of programmer time.

　　他們不能幫助程序員排除 bug 。這是一個簡單的成本/收益權衡?；ㄒ稽c時間簡化重現 bug 的過程或探索一下各種發生它的方法可以節約程序員大量的時間。

　　6. They are insulting, so they poison the relationship between developers and testers.

　　它們是侮辱性的，破壞了開發人員和測試人員的關系。

　　[Kaner93] has an excellent chapter (5) on how to write bug reports. Read it.

　　[Kaner93]有一章(第5章)非常好的內容說明了應該如何寫 bug 報告?？梢宰x一下。

　　Not all bug reports come from testers. Some come from customers. When that happens, it's common for a tester to write a regression test that reproduces the bug in the broken version of the product. When the bug is fixed, that test is used to check that it was fixed correctly.

　　不是所有的 bug 報告都是測試員寫的。有一些是來自客戶的。如果出現這樣的情況，常見情況是測試員編寫一個回歸測試，在產品出現問題的版本上重現這個 bug 。如果 bug 得到修復，這個測試可以用于檢查它是否正確修復。

　　However, adding only regression tests is not enough. A customer bug report suggests two things:

　　但是，僅僅添加回歸測試是不夠的?？蛻?bug 報告暗示著兩個東西：

　　1. That area of the product is buggy. It's well known that bugs tend to cluster.

　　產品的那個領域包含了 bug。大家都知道， bug 一般是集中出現的。

　　2. That area of the product was inadequately tested. Otherwise, why did the bug originally escape testing?

　　產品的那個領域沒有進行充分測試。否則的話，為什么開始測試的時候漏掉了那個 bug ?

　　An appropriate response to several customer bug reports in an area is to schedule more thorough testing for that area. Begin by examining the current tests (if they're understandable) to determine their systematic weaknesses.

　　對于某個領域中的幾個客戶 bug 報告的適當響應是對該領域安排一個更全面的測試。首先檢查當前測試(如果它們是可以理解的話)以確定在系統性方面的不足之處。

　　Finally, every bug report is a gift from a customer that tells you how to test better in the future. A common mistake is failing to take notes for the next testing effort. The next product will be somewhat like this one, the bugs will be somewhat like these, and the tests useful in finding those bugs will also be somewhat like the ones you just ran. Mental notes are easy to forget, and they're hard to hand to a new tester. Writing is a wonderful human invention: use it. Both [Kaner93] and [Marick95] describe formats for archiving test information, and both contain general-purpose examples.

　　總之，每個 bug 報告都是客戶的禮物，告訴我們在今后如何更好地測試。一個常見錯誤是不能為下次測試工作做好記錄。下一個產品將在某種程度上類似于這一個， bug 在某種程度上類似于這一個，你剛才所做的測試在某種程度上類似于將來找出那些錯誤的測試。腦海中的記錄容易忘記，也很難傳授給新測試員。書寫是人類一個美妙的發明：使用它。[Kaner93]和[Marick95]都描述了歸檔測試信息的格式，并包含了通用的示例。

原文轉自：http://www.kjueaiud.com