What is Code Coverage, and its significance?

The code coverage is the measurement of how much the product code has been exercised with all unit/functional/system tests.

A typical code coverage tool would produce metrics like how many or % of Java classes covered, methods covered, loops and instructions covered.

The significance of code coverage is to help in identifying the gaps in testing and also to assess the test coverage. Also, 100% code coverage should not be viewed as a high quality of the product. Instead, the results can trigger some more testing or optimization of the code.

Many times, code coverage is done after fully tested the product to see if any dead code or to find the missed testing coverage. Most of the times, the code coverage can be taken at the class level but at least method level is important. Usually, the QE team will report code coverage reports on regular basis.

The QA engineer should co-ordinate with all relevant test teams in carrying out such effort. The major tasks include the understanding of the tool and automation of the enabling the tool with the installed product, gathering the results, merging of results from different runs and publishing of the collective report. The report will be distributed to the stakeholders by a Java package name or by a module. Each module-level QE engineer can see if the results are appropriately covered or not and add the tests if required to cover all the classes and methods in that module. The feedback should be taken into the developer team to add unit tests or remove the dead or unused code for the product. Usually above 40-50% code coverage is considered as high coverage.

Why not 100% coverage target? It is all about human resources invested for testing of the product, and thereby 100% means that enormous test engineering resources requirement and practically is not feasible. Achieving higher code coverage can be aimed for any product as part of the quality testing of the product.

Some of the free and open source code coverage tools are

  • JaCoCo (nowadays more popular and originally developed from Emma)
  • JCov
  • Emma
  • Cobertura
  • Clover
  • PIT

What is Quality Assurance and its Role?

In the software industry, the quality related professional role is called as QA (Quality Assurance) or QE (Quality Engineering) engineer. In general, the “Quality Assurance” and “Quality Engineering” roles carry out the similar effort. The QA is a traditional role originated from industrial manufacturing assembly where the final product quality tested by verifying against a checklist, inspected for any damages and finally put a quality sticker. The similar QA role came to IT Software but associated with Engineering background. So, QE role makes sense for the Software industry, and it stresses more on the importance of applying the Engineering principles rather than simply repeating the quality assurance actions.

The QA role involves the tasks such as understanding of the software product features & domain knowledge, writing test plans, writing test specifications, manual execution of tests and interpreting results. Later, one has to do the development of automated tests, automated execution, and generation of final results summary for release level regression testing. Note that the QA role need average coding skills, but it takes more time and energy than in general as expected by many people. Because, for each product release in the product line, the developed tests should be continuously executed and verified to make sure no regressions at any point in time. The Quality Engineering skills have to be applied all the time during the product development. The quality has to be measured, assessed and concluded to prove that product is meeting the customer expectations. Without the engineering principles, quality testing might be happening and can discover some bugs but it may not ensure the proper proof to justify the on the quality and might leave some gaps. In summary, apply the core engineering principles such as measuring, making consistency, repeatability, and correctness during the application of testing techniques.

It is difficult to do the above principles merely with the manual approach because of limitations such as human errors in producing consistent actions & output, testing coverage with quick, agile product development cycles and making sure each build is verified for regressions. The Quality Automation is one of the important things to address this continuous regression testing. The automation effort needs programming skills in the same software domain and relevant skills to achieve the high automation, which saves time and test coverage.

Sharing my book that covers 16 core skills for QA & Java Automation professionals. Product Details Software Quality and Java Automation Engineer Survival Guide: Basic Concepts, Self Review, Interview Preparation (500+ Questions & Answers)Nov 16, 2016 by Jagadesh Babu Munta (More buying options such as iBookStore/Lulu/eBay can be found at http://www.everydayon.com ). Also, the paper book is available in India at http://www.everydayon.in

The below are 16 core skills covered in this book along with hands-on/getting started on a new job.

At a high level, this book starts with the areas such as processes (SDLC/Quality/OOPS), data (XML/XPath/JSON), code repositories (SVN/GIT), and operating systems (Unix/Linux). Then enters into programming (Java&JDBC), and later build frameworks/tools (ANT/Maven). After that, it covers more on unit testing frameworks/tools (Junit/TestNG) and continuous integration tools (Jenkins/Hudson). Next, it includes the functional testing of web applications and web services with frameworks/tools (TestNG/Selenium/SoapUI) covered. Finally, included samples code (for Java/Selenium) to reference offline, and also added hands-on/getting started in the job.

The preview of the book is available at https://drive.google.com/open?id=0BxzQzD6oofWmWjlVM0VsUTVpOU0

What are advantages and disadvantages of Agile Scrum process?

Scrum process advantages

  • Adapt to the changes where customer requirements can be honored in the middle of the project development (that is in the next sprint cycle).
  • It helps in doing incremental product delivery.
  • It takes less time comparing with waterfall model.
  • It gives an early indication of the project status and can be stopped quickly or can be transformed to a different project.

Scrum process disadvantages

  • It can consume more resources because sprint cycles need to cover all dev/test/doc activities at the same time.
  • Typically, it is good for shorter projects with less complex projects (where fewer alternatives/architectures to be considered).
  • It demands high productivity and constant effort be done by the team and thereby stress or time pressure felt by the team.
  • Also, it will have more quick sync’ up meetings.

What is Agile process?

The Agile Scrum is an iterative process of developing software with the goal of getting incremental intermediate product delivery. In the agile scrum process, there will be a product backlog, Sprint backlog and short cycles of the development process(typically seven days length of a sprint with the goal of delivering product in 4-5 weeks) and release. Sprints are repeated until entire product backlog is completed. Every release is towards an improvement or finishing towards the end product. Customer requirements can be injected into product backlog and will be picked up by next sprint (through sprint backlog). There will be daily stand-up meetings of approx. 15 mins in length on the scrum sync’ up.

How to understand the Waterfall development process?

The Waterfall is the traditional development model and as the name indicates, it is like water flowing from upper ground to next lower ground where development (along with information) passes through different phases as one after the other. It involves five different phases in an order and goes to next phase only after completing the prior phase.

The below are the major phases included in the process (with the same order)  –

i) Inception

ii) Design

iii) Implementation

iv) Verification

v) Maintenance.

Waterfall model advantages:

  • It can give better project schedule estimation and planning.
  • It provides the re-use of Resources because of sequential phases.
  • Good for large projects with clear planning, design, architecture before actual implementation. The team is not under time pressure (in comparison with Agile) as there can be sufficient time buffer for planned tasks. Typical project release duration is higher and is between 6 months to 1-5 years compared to Agile monthly project releases.

Waterfall model disadvantages:

  • Customer requirements can’t be accommodated in the middle of the project.
  • This takes more time and incurs more cost.
  • There is no incremental product delivery. The final project is available only after the final phase, which is a longer in duration.
  • It is difficult to predict if the project is doomed to failure until the project completion or out of resources.

How to differentiate White box vs. Black box testing?

White box testing

  • White box or unit testing is nothing but testing by looking at the actual or product code and focus on the units of the code to verify for basic code health.
  • It is also called Unit testing.
  • Typically performed by product developers.
  • Example test frameworks used are JUnit, TestNG for Java and NUnit family frameworks.

Black box testing

  • Black box testing is nothing but testing of the product without looking at the developed code and focuses on the customer functional use cases or scenarios.
  • It is also called functional testing.
  • Typically performed by QA team
  • Example test frameworks used are TestNG, Selenium, Load Runner, etc.

 

[Visit www.everydayon.com or (India)www.everydayon.in]

What is Regression Testing?

Regression testing is the execution of already created test suite or executing manual scenarios again. The goal is to make sure that already-existing features continuing to work while code changes are happening. Typically, these tests executed on hourly, daily, weekly or bi-weekly on different platforms and test matrix based on the product release cycles. Also performed regression testing at the time of build creation and developer check-ins.

Can we eliminate all bugs in a given product through extensive testing? What should we do?

No. Because of the complexity of software programming algorithms and code, it is impossible to eliminate all the bugs in a given product. It would need unlimited resources, time and even then no guarantee for zero bugs.

Thereby, typically quality is assessed as per planned criteria and customers can view the quality at different levels such as low, medium and high.

 

What is a Software bug?

A bug is nothing but an unexpected or deviation from expected behavior in the product. It could be detected during either positive or negative testing. That means either no errors or exceptions or missing things in the positive cases or as expected errors during the negative cases where the user should receive it. Bugs are recorded with all the relevant information to reproduce the observations or test scenario in a bug tracking software like bugzilla/JIRA tool.

What is a Test Plan? Any Template to use?

 

The test plan is a testing artifact in the planning process to plan the testing with details like scoped, non-scoped requirements to test, test strategy, resources, matrix and schedule (estimates) created by the QA team.

At a high level, there are two types of plans based on the scope on the project/module level.

  1. Master test plan, which is for the entire product/project.
  2. Module test plan, which is for the individual modules.

Master test plan is based on Market requirements document (MRD)/Business requirements document (BRD)/product requirements document (PRD)/Architecture & Design document.

Module test plan is similar but focused on a module/component under test based on the Master test plan and component functional specification.

A typical module test plan content is shown as below. You can take this as a template for the test plan even though some variations might exist in the organizations.

Introduction or Overview

  • Add brief introduction on the component or module to be tested.

Objectives

  • High-level ideas on the component testing goals.

Features Covered (in Scope)

  • High-level list of feature names to be tested
  • Prioritized features

 

Features NOT Covered (relevant but not in scope)

  • Certain nonfunctional features like any other team is going to cover such as performance; other testing can’t be covered such as localization (l10n) or internationalization (i18n).

Test Methodology

  • Framework selection
  • Tools to be used
  • Manual or Automation testing strategy
  • Regression testing strategy
  • Bugs – tools/infrastructure details
  • Security testing
  • Code coverage

Entry Criteria

  • Add conditions that are needed to be met before testing can start.
  • NOTE: Usually this will be covered in the Master test plan (MTP) and add the variations only here and refer others to MTP.

Exit Criteria

  • Add conditions that need be fulfilled for signoff, like bugs (severity) to be deferred, %of pass tests,
  • Note: Usually this will be covered in the Master test plan.

Test Certification Matrix

  • Note: Usually this will be covered in the Master test plan and add the variations only here and refer others to MTP.
  • Add Platforms/OS/JDK/Browser types and versions and any other software needed for the certification.

Schedule, Responsibilities, Deliverables

  • Schedule
    • Task|Owner|Duration|Start Date|End Date|Comments
  • Responsibilities
    • Team’s individual owned areas
  • Deliverables
    • TS/TDS (Test Spec/Test Design Spec)
    • New Tests
    • Automation scripts
    • CI setup
    • Results Analysis

Risks and Mitigations

  • What are the possible risks – like delay in builds can cause testing delayed, features integration late can cause late test cycles etc.
  • Any assumptions should be added here.

Reviewers and Approvals

  • Add Development counter-parts(corresponding dev engineers), Leads, Managers

References

  • List of referenced docs – like PRD, Design doc, Functional specs, Master test plan doc etc.

 

Revision History

  • Maintain the history of high-level changes on the test plan to track.

 

[Visit www.everydayon.com or (India)www.everydayon.in]