What is meant by a Test scenario? Where do you put test scenarios?

The Test scenario is nothing but a sequence of steps to be performed with a goal of verifying a user story or use case or requirement. These scenarios are going to become the test cases or tests and grouped as test suites.

Test scenarios are going to be created during the test design process and will be documented in the test specification.


What is Release Criteria? Who sets this up and who has to follow?

The Release criteria is the minimum checklist or list of goals to be achieved to release the product. This effort is driven by RE team, and all the project stakeholders would contribute to making this release to happen.

Below is a sample release criteria/checklist and can be taken as a template if none is available.

  • Bugs criteria

    • No P1, P2, P3 open bugs
  • Quality criteria

    • 100% feature coverage and test development
    • 100% test execution
    • 95-100% pass rate
    • >90% automation for regressions
  • Stress/Load/Concurrency criteria

    • 100 users/sec
  • Performance criteria

    • Minimum throughput numbers related to the product.
    • Comparison to other competitive products
  • Code coverage

    • >90% class level
    • >50% method level
    • >40% instruction/statement level
  • Internationalization (I18N) & Localization (L10N)

    • Internationalization support
    • Specific languages to translate
    • NOTE: I18N is short form as there are 18 characters between I and N in the “InternationalizatioN”. Similarly, L10N is short form as there are 10 characters between L and N in the “LocalizatioN”.
  • Documentation (books/online)

    • Release notes
    • User Manuals
      • Installation Guide, Admin Guide, Developer Guide, Troubleshooting guide, etc.
    • License and license text
    • Support/Maintenance plan

What is the Test development process? What are key steps?

The Test development process is the part of SDLC with strategy, methodology, frameworks and tools, etc. to be applied in the product testing.

The below are the key steps involved in the process.

First is to understand the component/module or integration of the product functionality expected behavior. It means that one should go through the use cases/user stories/features and discuss with developers for any clarifications. For this, use functional specification or JIRA tickets for user stories or tasks.

Second is to think and brainstorm with the team about all test scenarios for a particular use case. That means to create the exact sequence of steps to be performed for each use case as if like testing manually. Put of those scenarios incrementally in the Test Spec (TS) based on agile process or waterfall SDLC process.

The Third is to execute the tests manually and file bugs if any deviation from the expected behavior. In summary, for all test scenarios (based on priority) from TS

  • Run it manually if possible (example: going through web pages navigation/flow)
  • Write the test code/test script
    • Typically, it is done in the same language that code has been developed, like for java project is java based test code.

Fourth is to create a Test Development Framework, which is a set of utilities/tools to help in the following tasks:

  • Execution engine/driver to trigger test cases runtime/execution. Otherwise one has to run manually, say using simple java command line.
  • Supply input data as properties or CSV or other data files. Otherwise, one has to run manual supplying of arguments.
  • Generate Result/Report document (say HTML or XML). Otherwise, one has to look at the console for results.

Fifth is to select or create a custom Test Tool/Driver

  • Identify and select an open source/free/commercial right tool that helps in the testing or automation of tests. For example, Selenium for web site test automation or SoapUI tool for web services testing.

Six is to design and implement a Reporting mechanism

  • Design the final reports like TestNG or any customer HTML reports with basic system under test details, number of tests executed, tests passed, tests failed, tests skipped, etc.

Seventh is to setup SUT (System Under Test)

Eighth is to run test scenarios manually and file bugs if any.

Ninth is to automate the tests for regression testing.

  • Create the test scripts/test code along with a base build framework (as dev used in their builds) like Ant or Maven or Gradle build tools.
  • Review the test code with team (peers/dev).
  • Check-in the scripts/code into SCM repository (say git and along with product code).

Finally, perform regression testing

  • Execute the automated tests build by the build of the product.
  • Schedule automated execution jobs using tools such as Jenkins/Hudson CI.
  • Send the daily test result reports
  • Analyze the failures
  • File regression bugs
  • Verify and close bugs

What is Code Coverage, and its significance?

The code coverage is the measurement of how much the product code has been exercised with all unit/functional/system tests.

A typical code coverage tool would produce metrics like how many or % of Java classes covered, methods covered, loops and instructions covered.

The significance of code coverage is to help in identifying the gaps in testing and also to assess the test coverage. Also, 100% code coverage should not be viewed as a high quality of the product. Instead, the results can trigger some more testing or optimization of the code.

Many times, code coverage is done after fully tested the product to see if any dead code or to find the missed testing coverage. Most of the times, the code coverage can be taken at the class level but at least method level is important. Usually, the QE team will report code coverage reports on regular basis.

The QA engineer should co-ordinate with all relevant test teams in carrying out such effort. The major tasks include the understanding of the tool and automation of the enabling the tool with the installed product, gathering the results, merging of results from different runs and publishing of the collective report. The report will be distributed to the stakeholders by a Java package name or by a module. Each module-level QE engineer can see if the results are appropriately covered or not and add the tests if required to cover all the classes and methods in that module. The feedback should be taken into the developer team to add unit tests or remove the dead or unused code for the product. Usually above 40-50% code coverage is considered as high coverage.

Why not 100% coverage target? It is all about human resources invested for testing of the product, and thereby 100% means that enormous test engineering resources requirement and practically is not feasible. Achieving higher code coverage can be aimed for any product as part of the quality testing of the product.

Some of the free and open source code coverage tools are

  • JaCoCo (nowadays more popular and originally developed from Emma)
  • JCov
  • Emma
  • Cobertura
  • Clover
  • PIT

What is Quality Assurance and its Role?

In the software industry, the quality related professional role is called as QA (Quality Assurance) or QE (Quality Engineering) engineer. In general, the “Quality Assurance” and “Quality Engineering” roles carry out the similar effort. The QA is a traditional role originated from industrial manufacturing assembly where the final product quality tested by verifying against a checklist, inspected for any damages and finally put a quality sticker. The similar QA role came to IT Software but associated with Engineering background. So, QE role makes sense for the Software industry, and it stresses more on the importance of applying the Engineering principles rather than simply repeating the quality assurance actions.

The QA role involves the tasks such as understanding of the software product features & domain knowledge, writing test plans, writing test specifications, manual execution of tests and interpreting results. Later, one has to do the development of automated tests, automated execution, and generation of final results summary for release level regression testing. Note that the QA role need average coding skills, but it takes more time and energy than in general as expected by many people. Because, for each product release in the product line, the developed tests should be continuously executed and verified to make sure no regressions at any point in time. The Quality Engineering skills have to be applied all the time during the product development. The quality has to be measured, assessed and concluded to prove that product is meeting the customer expectations. Without the engineering principles, quality testing might be happening and can discover some bugs but it may not ensure the proper proof to justify the on the quality and might leave some gaps. In summary, apply the core engineering principles such as measuring, making consistency, repeatability, and correctness during the application of testing techniques.

It is difficult to do the above principles merely with the manual approach because of limitations such as human errors in producing consistent actions & output, testing coverage with quick, agile product development cycles and making sure each build is verified for regressions. The Quality Automation is one of the important things to address this continuous regression testing. The automation effort needs programming skills in the same software domain and relevant skills to achieve the high automation, which saves time and test coverage.

Sharing my book that covers 16 core skills for QA & Java Automation professionals. Product Details Software Quality and Java Automation Engineer Survival Guide: Basic Concepts, Self Review, Interview Preparation (500+ Questions & Answers)Nov 16, 2016 by Jagadesh Babu Munta (More buying options such as iBookStore/Lulu/eBay can be found at http://www.everydayon.com ). Also, the paper book is available in India at http://www.everydayon.in

The below are 16 core skills covered in this book along with hands-on/getting started on a new job.

At a high level, this book starts with the areas such as processes (SDLC/Quality/OOPS), data (XML/XPath/JSON), code repositories (SVN/GIT), and operating systems (Unix/Linux). Then enters into programming (Java&JDBC), and later build frameworks/tools (ANT/Maven). After that, it covers more on unit testing frameworks/tools (Junit/TestNG) and continuous integration tools (Jenkins/Hudson). Next, it includes the functional testing of web applications and web services with frameworks/tools (TestNG/Selenium/SoapUI) covered. Finally, included samples code (for Java/Selenium) to reference offline, and also added hands-on/getting started in the job.

The preview of the book is available at https://drive.google.com/open?id=0BxzQzD6oofWmWjlVM0VsUTVpOU0

What are advantages and disadvantages of Agile Scrum process?

Scrum process advantages

  • Adapt to the changes where customer requirements can be honored in the middle of the project development (that is in the next sprint cycle).
  • It helps in doing incremental product delivery.
  • It takes less time comparing with waterfall model.
  • It gives an early indication of the project status and can be stopped quickly or can be transformed to a different project.

Scrum process disadvantages

  • It can consume more resources because sprint cycles need to cover all dev/test/doc activities at the same time.
  • Typically, it is good for shorter projects with less complex projects (where fewer alternatives/architectures to be considered).
  • It demands high productivity and constant effort be done by the team and thereby stress or time pressure felt by the team.
  • Also, it will have more quick sync’ up meetings.

What is Agile process?

The Agile Scrum is an iterative process of developing software with the goal of getting incremental intermediate product delivery. In the agile scrum process, there will be a product backlog, Sprint backlog and short cycles of the development process(typically seven days length of a sprint with the goal of delivering product in 4-5 weeks) and release. Sprints are repeated until entire product backlog is completed. Every release is towards an improvement or finishing towards the end product. Customer requirements can be injected into product backlog and will be picked up by next sprint (through sprint backlog). There will be daily stand-up meetings of approx. 15 mins in length on the scrum sync’ up.