Validation sample size calculator

Using the ASQA sample size calculator

The calculator will ask you to input three figures:

• the number of assessment judgements
• your estimated error level, and
• your confidence level.

Both error level and confidence level have been set to default values.

1. Number of assessment judgements

This is the total number of assessment judgements made in the training product you are validating within a set period of time.

Determine the period of time to be considered based on your training and assessment strategies and timetabling. Consider assessment judgements made over a period of at least six months; this aligns with the retention requirements described in ASQA’s Guidance for Providers—Retention requirements for completed student assessment items.

For example, if your RTO has assessed 150 learners against the requirements of a unit of competency in the last six months, you would enter ‘150’ as the number of assessment judgements.

2. Error level

The error level relates to the assessment outcome results. In sampling terms, the error level can also be referred to as the ‘margin of error’.

To determine an appropriate sample size, you need to consider how likely it is that the sample assessment outcomes will be a good representation of the total assessment outcomes. 

When most assessment judgement outcomes are the same (for example, competent after the first assessment attempt), it is more likely that the sampled assessments will accurately represent all assessments. The error level, or risk that the validation sample size will not represent the total assessments, is not high. The recommended/default error level for this scenario is 15%.

However, if your assessment outcomes are varied—for example a mix of competent, competent after multiple attempts, and not competent assessment outcomes— there is a greater chance that the sampled assessments will not reflect the total assessments. To decrease this risk, decrease the error level. Note that when you lower the error level, the sample size will increase. Lowering the error level to any variant between 10 per cent and 15 per cent will allow for a good representation of the total assessment outcome results.

3. Confidence level

The confidence level relates to the assessment judgements.

The confidence level is how sure you need to be that the sample assessment judgements will produce an accurate validation outcome.

When there are risks that the validation outcome of a sample of assessments will not be reflective of the total assessments, you need to increase the confidence level.

When there are similarities in the factors that influence the assessment judgements, it can be assumed that validation of a smaller sample of these assessments will still produce an outcome which is representative of all assessment judgements. This can include, for example, when all assessment judgements have been made by one assessor or have been made by a number of assessors whose work has previously been validated.

If you perceive there is a higher risk that the validation outcome will not be reflective of all assessment judgements, for example when a newly-qualified assessor has made the judgements, you should increase the confidence level. Increasing the confidence level increases your chance that the sample will be statistically valid.

In sampling, the generally accepted confidence level is 95%. If you want to increase the confidence level, and obtain a larger sample size, any variation between 95% and 99% is appropriate.

Using ASQA’s validation calculator

Watch this video for a quick demonstration of how to use ASQA’s validation calculator, which helps you determine the required sample size.