automatic testing - challenges

"practice of testing .... turned our art into engineering, introduced process-models, come up with best-practices, and developed tools ..."

Google Testing Blog: Automating tests vs. test-automation

downsides, problemer:

• "Scripting your manual tests this way takes far longer than just executing them manually.
• The UI is one of the least stable interfaces of any system, so we can start automating quite late in the development phase.
• Maintenance of the tests takes a significant amount of time.
• Execution is slow, and sometimes cumbersome.
• Tests become flaky.
• Tests break for the wrong reasons."

men: "advantages of automation still outweigh the cost. .... accept some of these problems as 'the price of automation'"

og der er workarounds:

• "It takes long to automate a test—Well, let's automate only tests that are important, and will be executed again and again in regression testing.
• Execution might be slow, but it is still faster than manual testing.
• Tests cannot break for the wrong reason—When they break we found a bug."

nokre poeng frå artikkelen (mi utheving):

"...a system exposes different interfaces to the environment—e.g., the user-interface, an interface between front-end and back-end, an interface to a data-store, and interfaces to other systems—it is obvious that we need to look at each and every interface and test it. ... also avoid testing the functionality in too many different places."

"...how this system looks inside.

• Is there a database? If so, the verification should probably not be performed against the UI but against the database.
• Do we need to interface with a supplier? If so, how should this interaction look?
• Is the same functionality available via an API? If so, it should be tested through the API, and the UI should just be checked to interact with the API correctly."

"Applying these simple questions will allow us to:

• write many more tests through the API, e.g., to cover many boundary conditions, ....
  
• start earlier with testing the system, as we can test each interface when it becomes 'quasi-stable',
• makes maintenance of tests and debugging easier, as the tests break closer to the source of the problem, ...."

"To summarize, I figured out that a successful automation project needs:

• to take the internal details and exposed interface of the system under test into account,
• to have many fast tests for each interface (including the UI),
• to verify the functionality at the lowest possible level,
  
• to have a set of end-to-end tests,
• to start at the same time as development,
• to overcome traditional boundaries between development and testing (spatial, organizational and process boundaries), and
  
• to use the same tools as the development team."