As QA and development managers you might want to know the test plans success rate.
I've tried to use the "official" measurements for that, but the results were not accurate, for example - the Test Created when used in Test Plan. I found myself maintaining the report over and over again.
Assumption: I take it the data in the XRAY Test Plan itself is the accurate one.
The below report doesn't have any filters beside the test plan ID. You see the native measure Test Created different than the actual in the test plan itself (75 vs 82).
I tracked the missing tests, they are in the Jira XRAY Test Plan in the test list, and in the Test Execution in this Test Plan.
==========
This is another example, not a must read as I proved my poine above.
Just for the sake of it, this is an "official" measure to find the TODO tests numbers per Test Plan in this case:
CASE WHEN [Xray Test Run Status].CurrentMember IS [Xray Test Run Status].DefaultMember
THEN
([Measures].[Xray Tests created],
[Xray Test Run Status].[TODO])*1
END
You can see it here.
(in this case it is divided for percentage, also use an older measure: Xray Test Execution Status and not the updated one: Xray Test Run Status), but in principle it is the same).
It counts in the below case 75 pass, while another measure eazyBI support gives, counts the correct 82.
This is an alternative measure from eazyBI support:
CASE WHEN --only when test run statuses in the report Not IsEmpty([Measures].[Xray Test Plan Overall Execution Status]) AND [Xray Test Run Status].CurrentHierarchyMember.Level.Name = "Status" THEN --find status in summary matching Test Run Status name extract number following status name Cast( ExtractString( [Measures].[Xray Test Plan Overall Execution Status], [Xray Test Run Status].CurrentMember.GetCaption||": ([^,]*)",1 --set formatting to number so you could represent it in chart if needed ) AS NUMERIC) ELSE --when no statuses selected show property as it is [Measures].[Xray Test Plan Overall Execution Status] END
and what you see in the above pic as TP Overall Status Split.
In general, seems that the 'TP Overall Status Split' is accurate, since they use the same "trick" or source as I do: they use the measure Xray Test Plan Overall Execution Status. This measure just brings the execution status as is from the test plan as is.
I also use the measure Xray Test Plan Overall Execution Status, but I tackled it a little different, and with the ability to add percentages.
For example: Test-Run Pass
Val(
ExtractString([Xray Test Plan].CurrentHierarchyMember.get('Test Plan Status'), 'PASS:\s*(\d+)',1)
)
Measure set with Integer Formatting.
So I extract the Pass number from Xray Test Plan Overall Execution Status and set it to value. Same for all tests existing statuses. From here is is I can get the Success rate percentage:
% Pass from the number of tests that pass+fail:
CASE
WHEN CurrentTuple(VisibleRowsSet()).Item(0).Name <> '$total_aggregate'
THEN
CoalesceEmpty(
[Measures].[Test-Run Pass]/
(
[Measures].[Test-Run Pass]+[Measures].[Test-Run Failed]
)
, 0)
or % of tests that were executed (= Pass and failed):
CASE
WHEN CurrentTuple(VisibleRowsSet()).Item(0).Name <> '$total_aggregate'
THEN
CoalesceEmpty(
(
[Measures].[Test-Run Pass]+
[Measures].[Test-Run Failed]
)/
[Measures].[All Tests in TP]
,0)
ELSE
(
[Measures].[Sum TR Pass]+[Measures].[Sum TR Failed]
)/
[Measures].[Sum all TP]
END
The measure inside the above measure:
Sum of all TP:
SUM(
VisibleRowsSet(),
[Measures].[All Tests in TP]
)
Sum TR Pass
SUM(
VisibleRowsSet(),
[Measures].[Test-Run Pass]
)
Sum TR Failed
SUM(
VisibleRowsSet(),
[Measures].[Test-Run Failed]
)
Why the Case?
To get the correct percentage in the Total row.
The built-in measures Issue Test Plan Status and Xray Test Plan Overall Execution Status are the same.
I suspect that because the results were not consistent they changed the measure: Xray Test Plan Overall Execution Status
Comments
Post a Comment