TestabilityReid Holmes

http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf https://www.youtube.com/watch?v=1keyEiJHqPw https://www.youtube.com/watch?v=XcT4yYu_TTs

REID HOLMES - CPSC 410: Advanced Software Engineering

TestabilityThe degree to which a system or component facilitates the establishment of test objectives and the execution of tests to determine whether those objectives have been achieved.

REID HOLMES - CPSC 410: Advanced Software Engineering

Given a finite amount of time and resources, how can we validate that the system has an acceptable risk of costly or dangerous defects.

—Bob Binder

REID HOLMES - CPSC 410: Advanced Software Engineering

Develop

Test

Check-In

REID HOLMES - CPSC 410: Advanced Software Engineering

Why not test?‣ Good reasons:‣ I don’t know how!‣ Legacy code

‣ Bad reasons:‣ Bad design‣ Doesn’t catch bugs (now)‣ Slow‣ Boring‣ Hard to change‣ That’s QA’s job

REID HOLMES - CPSC 410: Advanced Software Engineering

Common assumption‣ “The cost of fixing faults rises exponentially with

how late [e.g., requirements, design, implementation, deployment] they are detected.”‣ This is commonly stated but is based on evidence

from 20+ years ago.‣ This assumption does not seem to hold for

modern processes, tools, and languages.‣ That said, it is still necessary to validate that the

system works.‣ Also important that it continues to work as the

system evolves.

REID HOLMES - CPSC 410: Advanced Software Engineering

Terminology‣ Efficiency: number of tests per unit of effort.‣ Effectiveness: the probability of detecting a bug per

unit of effort.‣ Higher testability: more effective tests, same cost.‣ Lower testability: fewer weaker tests, same cost.

‣ Repeatability: the likelihood that running the same test twice will yield the same result.

‣ SUT/CUT: System/Code Under Test‣ White-box: tests consider internals of CUT.‣ Black-box: tests are oblivious of internals of CUT.

http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf

REID HOLMES - CPSC 410: Advanced Software Engineering

Anatomy of a test‣ To reveal a fault, a test must:‣ Reach some code‣ Trigger a defect‣ Propagate an incorrect result‣ The result must be observed‣ The result must be interpreted as incorrect

‣ Test threats:‣ Non-deterministic dependencies‣ Threading/Race Conditions/Deadlock‣ Shared data

REID HOLMES - CPSC 410: Advanced Software Engineering

Properties of Testability‣ Controllability‣ The extent to which the SUT can be made to

perform specific actions of interest.‣ Observability‣ The extent to which the response of the SUT to a

test can be verified.‣ Isolateability‣ The degree to which the element under test can

be validated on its own.‣ Automatability‣ The ability to execute the test programmatically.

REID HOLMES - CPSC 410: Advanced Software Engineering

Controllability‣ What do we have to do to run a test?‣ How expensive is it?‣ Does the SUT make running a test impractical?‣ Give a test goal, do we have enough information to

create an adequate suite?‣ How much tooling can we afford?

REID HOLMES - CPSC 410: Advanced Software Engineering

Observability‣ What do we have to do to identify pass/fail?‣ How expensive is it to do this?‣ Can we extract the result from the SUT?‣ Do we know enough to identify pass/fail?

REID HOLMES - CPSC 410: Advanced Software Engineering

Isolateability‣ Can the element being tested be isolated?‣ What is the cost to do this?‣ If an element cannot be naturally isolated, can we

simulate it (e.g., with mocks / stubs)?‣ Why bother?‣ Isolated components are:‣ Simpler to reason about (e.g., root cause analysis)‣ Less prone to non-determinism‣ Faster

‣ Simulated dependencies can also more enable validating unusual states.

REID HOLMES - CPSC 410: Advanced Software Engineering

Automatability‣ Can tests be executed without human intervention?‣ Huge economic advantages:‣ Setting up automation: 5 hours‣ Running manual test: 30 minutes.‣ Automation pays for itself after just 10 iterations.

‣ What is the cost of automated infrastructure?‣ What is the benefits of using a test infrastructure?‣ Executions can be batched.‣ Run on same configuration / hardware.‣ Global visibility of results.

‣ Enables regression testing.

REID HOLMES - CPSC 410: Advanced Software Engineering

Challenges‣ Tests are code too.‣ Also subject to their own faults.

‣ Not all test failures uncover faults:‣ Defect in test itself‣ Flaky test (due to some form of non-determinism)‣ Requirements shortcoming (undefined behaviour)‣ Implicit assumptions often surfaced by tests

‣ How to retrospectively recognize a ‘true’ failure?‣ Developer changed the source (not test), test

passed on the next iteration.

REID HOLMES - CPSC 410: Advanced Software Engineering

Develop

Test

Check-In

Must: Be Fast Be Reliable Isolate Failures

REID HOLMES - CPSC 410: Advanced Software Engineering

Kinds of tests

REID HOLMES - CPSC 410: Advanced Software Engineering

Test value

Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you onlyneed to search that small unit under test to find the bug. 

Writing effective unit tests requires skills in areas such as dependency management, mocking, and hermetic testing. Iwon't cover these skills here, but as a start, the typical example offered to new Googlers (or Nooglers) is how Googlebuilds and tests a stopwatch.

With end­to­end tests, you have to wait: first for the entire product to be built, then for it to be deployed, and finally forall end­to­end tests to run. When the tests do run, flaky tests tend to be a fact of life. And even if a test finds a bug, thatbug could be anywhere in the product.

Although end­to­end tests do a better job of simulating real user scenarios, this advantage quickly becomesoutweighed by all the disadvantages of the end­to­end feedback loop: 

Unit End­toEnd

Fast

Reliable

Isolates Failures

Simulates a Real User

Unit tests do have one major disadvantage: even if the units work well in isolation, you do not know if they work welltogether. But even then, you do not necessarily need end­to­end tests. For that, you can use an integration test. Anintegration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that theycoherently work together.

If two units do not integrate properly, why write an end­to­end test when you can write a much smaller, more focusedintegration test that will detect the same bug? While you do need to think larger, you only need to think a little larger toverify that units work together.

Even with both unit tests and integration tests, you probably still will want a small number of end­to­end tests to verifythe system as a whole. To find the right balance between all three test types, the best visual aid to use is the testingpyramid. Here is a simplified version of the testing pyramid from the opening keynote of the 2014 Google TestAutomation Conference: 

Unit Tests vs. End­to­End Tests

Integration Tests

Testing Pyramid

Kirkland (1)

Kurt Alfred Kluever (1)

Mark Ivey (1)

Mark Striebeck (1)

Marko Ivanković (1)

Michael Bachman (1)

Mike Wacker (1)

Mona El Mahdy (1)

Noel Yap (1)

Patricia Legaspi (1)

Peter Arrenbrecht (1)

Phil Rollet (1)

Pooja Gupta (1)

Radoslav Vasilev (1)

Rajat Dewan (1)

Rajat Jain (1)

Rich Martin (1)

Richard Bustamante (1)

Roshan Sembacuttiaratchy (1)

Ruslan Khamitov (1)

Sean Jordan (1)

Sharon Zhou (1)

Stephen Ng (1)

Tejas Shah (1)

Test Analytics (1)

Vojta Jína (1)

October 2015 (1)

August 2015 (1)

June 2015 (1)

May 2015 (2)

April 2015 (2)

March 2015 (1)

February 2015 (1)

January 2015 (2)

December 2014 (2)

November 2014 (1)

October 2014 (2)

September 2014 (2)

August 2014 (2)

July 2014 (3)

June 2014 (3)

May 2014 (2)

April 2014 (2)

Archives

This site uses cookies from Google to deliver its services, to personaliseads and to analyse traffic. Information about your use of this site isshared with Google. By using this site, you agree to its use of cookies.

LEARN MORE GOT IT

REID HOLMES - CPSC 410: Advanced Software Engineering

Test pyramid

http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html

REID HOLMES - CPSC 410: Advanced Software Engineering

Continuous Integration‣ Every project will build and execute automated tests.‣ TravisCI (https://travis-ci.com/ (NOT ORG)) has

provided the class with large numbers of commits‣ You can request access online, we will grant it

REID HOLMES - CPSC 410: Advanced Software Engineering

Activity‣ In your groups:‣ Choose one of your use cases‣ Describe one end to end test for this use case‣ Describe 2 integration tests for this use case‣ Describe 3 unit tests for this use case