iOS Automation

Code Coverage with Calabash

17 March 2016

Even though XCode7 is deprecating GCov output, it is still possible to generate .gcda and .gcno files for UI Automation. However, it is my experience that code coverage files are only generated for simulator. This means it won’t be possible to get code coverage for any tests that are limited to physical devices. (If you do find a way to get it to generate on a device, please let me know!)

It is important to note that you will get less accurate code coverage results with this deprecated gcov output from XCode7 than running tests with XCode6. It will, however, give you a minimum number. I am actively working to get the numbers closer and will report back.

Here are the steps you would follow...

Before the run (one time):

  1. Clean up from your previous run. Delete everything in your output directory.
  2. Generate a baseline file.

Each time you run a test case (scenario):

  1. Remove old .gcda files (code coverage results from the previous run).
  2. Run the test case.
  3. Flush (generate those .gcda files).
  4. Process all the .gcda files into a single .info file.

After all the tests are complete (one time):

  1. Combine all the .info files into a single file.
  2. Generate a report from that file using lcov.


Install this:

Homebrew

If you don’t have it, it’s a great way to install tools without having to compile everything yourself /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

lcov

This tool is used to generate reports. Gcovr is another option, but it looked like the results weren’t as accurate. brew install lcov


Implementation

XCode

Environment Variables

// Activating this setting causes '.gcno' files to be produced that the gcov code-coverage utility can use to show program coverage.
GCC_GENERATE_TEST_COVERAGE_FILES = YES
 
// Activating this setting indicates that code should be added so program flow arcs are instrumented.
GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = YES
 
// Lets us know if we have coverage enabled so we can turn the __gcov_flush() lines on/off as appropriate and avoid build errors
GTM_CONFIGURATION_GCC_PREPROCESSOR_DEFINITIONS = COVERAGE_ENABLED=1


Flushing

I’ve seen some people have some success with adding __gcov_flush() to their AppDelegate class. I decided to manually flush so I had more control.

Flushing in your Calabash backdoor (recommended flush method)
- (NSString*)flushGCov:(NSString*)ignored
{
#if defined(COVERAGE_ENABLED)
	extern void __gcov_flush(void);
	__gcov_flush();
	
	return @"GCov Flushed";
#endif
	return @"GCov not Flushed";
}
Flushing on applicationDidEnterBackground (alternative flush method)
Use at your own risk!

You would place this before your @implementation:

#if defined(COVERAGE_ENABLED)
#include <stdio.h>
extern void __gcov_flush();
#endif

And you would add this to your applicationDidEnterBackground method:

#if defined(COVERAGE_ENABLED)
	__gcov_flush();
#endif


Ruby Code

This is how we have ours set up. You can implement it your own way if you’d like :). Gist with the Ruby code


Reports

When the run completes, assuming you have called the methods like I did, you should have a CodeCoverage folder that contains a lot of .info files (including a *combined.info file with all the other .info files merged into it) and an html report folder.