Aim: The primary aim of this research was to investigate the potential benefits of changing the way in which positional uncertainty caused by information latency is represented in an Australian Army navigation support system. An important design consideration in developing the alternative way of representing uncertainty was to convey information about information latency in a transparent way. Transparency is a factor that can impact on operator trust. In turn, trust can influence operator use and reliance on systems.

This study first involved the development of an alternative set of icons for conveying the position of an entity on the map as well as the latency of the information used to derive the position. The design approach sort to maximise transparency whilst also seeking to be consistent with Australian Defence Department guidelines for military symbology. A 3-level icon representation of latency was consequently developed. As time without an update passes, the icon system cycles through the levels of ‘status’ from ‘up-to-date’ to ‘unsure’ to ‘unknown’. These different stages represent the age or confidence in the information. The 3-level system includes an additional middle ‘unsure’ status not present in the 2-level system found in the extant Army navigation support system. 3 levels was believed to offer a sufficient level of differentiation about the quality of the information (i.e. how old it was) without providing the operator with superfluous distinctions that may overwhelm and reduce performance. The 3-level system was kept similar to the current 2-level system in terms of colour choice to ensure that the efficacy of the levels was the source of any significant differences.

Design: This study was a 2 (Graphical User Interface (GUI): 2-level, 3-level) x 3 (latency: low, medium, high) experimental within-subjects design where participants used a different GUI in each scenario. Order of GUI presentation was counterbalanced to minimise order effects. Both scenarios followed the same latency structure, which was 10 3-minute blocks each with a designated level of latency (low, medium or high). The baseline, or low condition, updated the icon every 6 seconds; the medium condition was every 45 seconds while the high was every 90 seconds.

Method: Study participants were 41 individuals aged 18-56 years (M = 32 years, SD = 9.86; 24% female). Participants completed the study using simulators driven by PC based Virtual BattleSpace 3 software (VBS3; Bohemia Interactive). Each simulator represented a black SUV with a ‘navigator’ (participant) and ‘driver’ (confederate) position. The ‘navigator’ had a Surface Pro 2 tablet device (Microsoft) with a simplified emulation of the Army’s navigation support system which featured an interactive map of the simulated island. The map featured icon representations of their vehicle and the neutral (green) aid tents requiring supply drops. Participants completed 2 scenarios that had a maximum time allotment of 30 minutes.

During the scenario, the participant experienced periods of increased latency where the icons would stop updating. This affected the icon of their vehicle in two ways. First there was a change to position where the icon representing the participant’s vehicle would stop tracking until it received an update. Second, there was a change to status where the icon would change state based on time without an update.

Trust was assessed during and after the completion of the scenarios. Participants were required to report on their trust in the status and the position of the icons in the system at given times within each scenario. Following completion of the scenario participants completed a modified form of the trust in automation scale. A variety of measures were used to assess moderating factors of human-automation trust. These included experience with similar technology and attitudes toward computers in general.

Results: When comparing the 2-level and 3-level systems, participants had significantly higher trust in the icons of the 3-level system both during and after the scenario. During the scenario, this difference was most prominent in the medium latency condition. Participants’ experience with GPS systems and paper maps as well as their self-confidence with computers and performance in the scenarios were all significantly related to trust.

Conclusion: In line with previous research this study highlights the importance of transparent systems to appropriately calibrate user trust as well as improving user task performance. More specifically, it emphasises the need to consider how uncertainty is visualised, particularly in terms of how latency is represented to engender greater and more appropriate levels of trust.

Aim: The One Defence Leadership Behaviours (1DLB) form a key part of strategic cultural reform for Australia’s Department of Defence arising from the First Principles Review (FPR). The 1DLB model emerged from interviews with the senior leadership group about their practice, conducted in support of the FPR process. In essence, the 1DLB is an applied qualitative model that may have value as the basis for a general model of Australian administrative leadership. The model uses seven values to characterise leadership behaviour in Defence; contributor, learner, accountability, risk manager, inclusive, team builder and innovator. Given its qualitative basis, the aim of this study is therefore to establish evidence as to whether the qualitative model can be seen quantitatively.

Design: An empirical measurement model was developed using data from the 2018 version of the annual YourSay series. The YourSay survey series includes a combination of legacy and novel individual- and group-referent evaluative items across a range of human resource practices outcomes (e.g. leadership, career management, fraud and ethics, attitude to reform and inclusivity) designed to measure organisational climate. Notably, the items were never designed nor intended to measure the 1DLB individually or in combination. A combination of exploratory and confirmatory factor analysis across the Australian Public Service, Navy, Army and Air Force was used to respond to the aim of the study.

Method: An invitation to participate in the 2018 YourSay survey was sent to all members of the Department of Defence (including the Reserves). A total of n=33,944 responses were recorded. After data cleaning (e.g. opt outs, nonsensical responses, removal of Reserves and multiple imputation of missing data), n=17,916 responses across the four Services were available for analysis. Each Service was randomly divided, with the hold out sample available for confirmatory analysis. Representativeness analysis indicates an under-representation of relatively junior staff (younger men) and over-representation of executive-level staff (e.g. Executives and mid-career Officers). All non-demographic items were included in a principal components analysis.

Results: The four principal components analyses yielded 21 factors, seven of which were interpretable as consistent with 1DLB constructs. Item loadings revealed minor differences across the four Services. Factors were finalised based on item loading consistency across the Services and interpretability, assessed using a reliability analysis. The final seven factor solution was used to measure Contributor, Learner/Innovator, Accountability (Supervisor), Accountability (Senior Leaders), Risk Manager, Teamwork and Inclusivity.

Conclusion: Empirical analysis of an instrument never designed to measure the 1DLB demonstrated that the qualitative model is reflected quantitatively. This suggests the 1DLB may have validity as a way of understanding leadership behaviours in Defence specifically, and the Australian public sector more broadly. While the analysis shows that there appears to be some basis to the 1DLB beyond a set of interviews with senior leaders in Defence, further work is needed to establish both the measurement model and the constructs. One strand of this work is to develop the theory underlying the 1DLB. A second strand is to operationalise the behaviours and their measurement through scale development using sound psychometric principles. A third strand is to test whether the model has validity and reliability outside the Defence context.

Background. The Australian Army has been a pioneer in the use of assessment centres, known as Officer Selection Boards (OSB), in the selection of Australian citizens to be trained as officers. Using psychometric tests, interviews, and behavioural activities that assess leader and group-interaction adeptness in a range of situations, the OSB process has been carefully developed and robustly defended over many decades. Nonetheless, the recent growth in community awareness of unconscious bias and desire for diversity triggered a review of the decision-making process of OSB members and specifically its definitions of risk and success, noting that the Australian Army is predominantly an Anglo-male organisation.

Aim. This study evaluated the provision of training for OSB members that included two elements: (1) information on unconscious bias and (2) a decision-support matrix that detailed how to assess unconscious bias and diversity using behavioural anchors of success and risk during the OSB process. The overall aim was to determine whether the additional training would yield a higher success rate for a more diverse range of OSB candidates.

Design. A quasi-experimental approach was applied to OSBs as they occurred in the normal selection process for entry to the Royal Military College or the Australian Defence Force Academy. All OSB members received training in the conduct of an OSB using negative selection principles, with some also receiving additional training in unconscious bias and the decision-support matrix. The independent variable consisted of the presence or absence of the additional training for unconscious bias, with the ratings of applicants by OSB members being the dependent variable. The topics of gender (specifically female) and age (specifically outside the age band of 18-35 years) were particularly targeted in the training as potential areas of unconscious bias, whilst ethnic background (defined as Australian born or overseas born) was not specifically discussed in the training or decision-support matrix as a “quasi-control” topic.

Method. The rated performance of 998 applicants at OSBs for full time officer training during 2015 and 2016 were included. During 2015, applicants were assessed by OSBs whose members had been trained in traditional OSB procedures only. By contrast, applicants during 2016 were assessed by OSBs whose members had received the additional training. The applicants in the 2015 and 2016 cohorts were compared for differences in their general aptitude, scholastic achievement, leadership experience, teamwork experience, and physical agility plus sporting background. No significant differences were found.

Results. A series of chi-square and t-tests showed that candidates who were assessed by an OSB whose members had completed the additional training were more likely to be rated as performing better on the OSB activities than those who were assessed by an OSB who had not received the additional training or decision-support matrix, regardless of their diversity. Females were likely to be rated as performing better across both years, but the positive ratings for both age and ethnic background improved with the additional training and decision-support matrix.

Conclusions. This study shows that the added training can have a positive impact on selection decisions. The specific articulation of risk and success for different diversity factors within the decision-support matrix may have made a key difference to all candidate assessments. Thus, in circumstances where blind recruiting is neither possible nor appropriate, better articulation of how to assess success / achievement and associated risk can “nudge” positive outcomes without having to completely overhaul selection processes. Application of these findings to other practical environments, and the limitations of this study, will be discussed.