scan, aka run_tests, is a powerful fastlane action which allows you to automate Unit Tests for iOS applications.
Using scan action is quite simple. For example, a simple configuration could look like the following:
scan(
workspace: "path/to/MyAwesomeApp.xcworkspace",
derived_data_path: "path/to/derived_data",
scheme: 'MyAwesomeApp',
output_directory: 'path/to/reports'
clean: true
)
Under the hood scan makes usage of default parameters that, if not fine tuned correctly, could increase the time to complete the action itself.
In small projects the increased time could not be an issue, but in large projects, if you want to make the automation faster, you could end up overloading your CI / CD environment.
To understand better if default parameters may increase the completion time for running tests, I’ve performed few experiments within a quite large project, a mix of Swift and Objective-C, using a MacBook Pro with an Apple M1 Pro chip.
First of all, scan action comes with the parameter skip_build which is set to false. Leaving the parameter with its default value is the same as the following xcodebuild command:
xcodebuild \
-workspace MyAwesomeApp.xcworkspace \
-scheme MyAwesomeApp \
-sdk iphonesimulator \
-destination 'platform=iOS Simulator,id=<redacted>' \
build test
After setting skip_build to true (why is build action needed?), the completion time for running tests has been decreased by up to 50% of the inital value.
scan(
workspace: "path/to/MyAwesomeApp.xcworkspace",
derived_data_path: "path/to/derived_data",
scheme: 'MyAwesomeApp',
output_directory: 'path/to/reports'
clean: true,
skip_build: true
)
This is a fantastic result but I’m not still satisfied.
Reading at the documentation, I’ve found another parameter called output_style which defines how the output should look like. By default (when not set), scan uses xcpretty to format the console output in a prettier way.
After setting output_style to 'raw' (disabling xcpretty), the completion time has been decreased by up to 90% of the inital value.
scan(
workspace: "path/to/MyAwesomeApp.xcworkspace",
derived_data_path: "path/to/derived_data",
scheme: 'MyAwesomeApp',
output_directory: 'path/to/reports'
clean: true,
skip_build: true,
output_style: 'raw'
)
This is amazing!
There is a gotcha disabling xcpretty, though: html or junit reports are not generated anymore.
Since not still satisfied, I’ve reread with care scan documentation and I’ve found two additional parameters: build_for_testing and test_without_building.
Unfortunately the documentation does not describe too much about them. But then I’ve found a WWDC 2016 video (unlisted on Apple Developer site) that explains the purpose of them.
build_for_testing is an xcodebuild action that:
- builds sources for testing
- outputs products in Derived Data
- generates an xctestrun file
test_without_building, instead, consumes the outputs of the previous action to run tests against the xctestrun file.
With this in mind I’ve tried another experiment splitting the scan action into two different parts. In this manner I was able to maintain html and junit report generated by scan.
scan(
workspace: "path/to/MyAwesomeApp.xcworkspace",
derived_data_path: "path/to/derived_data",
scheme: 'MyAwesomeApp',
clean: true,
skip_build: true,
output_style: 'raw',
build_for_testing: true
)
scan(
workspace: "path/to/MyAwesomeApp.xcworkspace",
derived_data_path: "path/to/derived_data",
scheme: 'MyAwesomeApp',
output_directory: 'path/to/reports'
clean: true,
skip_build: true,
test_without_building: true
)