Ability to force error code to zero for failed unit tests
When running SQL CI in Bamboo if any unit tests fail, SQLCI.exe returns non-zero which is interpreted as a failure code by Bamboo. This means that the next step which is to parse the unit test results won't run, and you wont' know what test failed.
Can we have a way of forcing a zero return code for failed unit tests?
As described in comments, this is now possible with the Powershell cmdlets. Please get in touch if you need this facility in the Bamboo addon.
-
Tugberk commented
With the new [Invoke-DlmDatabaseTests](https://documentation.red-gate.com/display/SR1/Invoke-DlmDatabaseTests), controlling the exit behavior is easier. The default behavior on failed tests is to warn rather than error. For example, the below script would not terminate with non-zero exit code:
```
$ErrorActionPreference = "Stop"$testResults = Invoke-DlmDatabaseTests "D:\Dev\tsqlt-test"
if($testResults.TotalFailures -gt 0) {
Write-Output "There are $($testResults.TotalFailures) test failures"
}
else {
Write-Output "There are no failures"
}
```If you want to exit on failure, you can exit manually:
```
$ErrorActionPreference = "Stop"$testResults = Invoke-DlmDatabaseTests "D:\Dev\tsqlt-test"
if($testResults.TotalFailures -gt 0) {
Write-Output "There are $($testResults.TotalFailures) test failures"
exit 1
}
else {
Write-Output "There are no failures"
}
``` -
Mark Snelling commented
It's all good and well saying that it's Bamboo's fault but that doesn't help me at all. Their default I think is to check the return code on any external executable, perhaps TC, TFS and Jenkins don't check this and I would consider that to be the wrong approach.
The crux of the problem is there isn't an option to use your Bamboo plugin with Bamboo Cloud which means I have to run SQLCI.exe as an external command from the Bamboo agent (we do this by creating a custom AMI with DLMAS).
I've been using your suggestion wrapping the command in a batch file but it's just bitten me because the path to SQLCI.exe had changed in your latest release. -
Jason Crease commented
We return a non-zero exit code so CI systems (and home-rolled systems) can easily detect if there were test failures. This is a very standard thing... I reckon Bamboo is at fault here. TeamCity, TFS and Jenkins don't have this problem!
Anyway, here's a few workarounds:
1) Use cmd /c with the & EXIT 0 command: cmd /c "sqlci.exe arg1 arg2 & EXIT 0" [BTW: officially it SHOULD be EXIT /B 0, but this behavior was apparently broken on XP, so now Windows supports both EXIT 0 and EXIT /B 0]
2) Invoke PowerShell to invoke the SQLCI executable:PowerShell "sqlci.exe arg1 arg2".
PowerShell will not relay the non-zero exit code - it will return an exit code of 0.
@Mark Snelling, out of interest, how are you using SQL CI with Bamboo Cloud? Did you check in the binaries?
Jason Crease (Application Engineer, Redgate Software)
-
Hmmm... Shame. I'm no expert at Bamboo so I'm not sure how this could work. Have you tried calling this from Powershell as this might be more versatile in terms of how it behaves with failure exit codes?
-
Mark Snelling commented
Thanks David but we're using Bamboo Cloud and that plugin isn't available on that platform I'm afraid.
-
Thanks for reporting the issue. I've emailed you a link to our Bamboo plug-in.