AAIR Forum 2021

Theme

Get Your AAIR Booster Shot

Subthemes

The 2021 AAIR Forum subthemes include:

• Impacts of COVID-19
• The evolving role of the institutional researcher
• Justifying the value of higher education and research
• Evidence-based strategic planning (and implementation)
• Benchmarking for continuous quality improvement
• Informing learning and teaching through evaluation and surveys
• Realising benefits – the role of institutional research in enterprise and transformation projects
• Compliance and Government reporting reform
• Business intelligence, analytics, visualisation and modelling – the foundations of IR

Special interest groups

• Business intelligence and analytics
• Load management and planning
• Government reporting
• Surveys and evaluation

Organising committee

For the virtual forum in 2021, the organising committee is the current AAIR Executive Committee:

• Stuart Terry, Otago Polytechnic (AAIR Forum Chair)
• Don Johnston, Southern Cross University
• Pam Rayner, Bond University
• Kathie Rabel, Life Member
• Greg Jakob, Life Member
• Lisa Bolton, The Social Research Centre
• Andrew Bradshaw, Macquarie University
• Leone Nurbasari, The Australian National University
• Eva Seidel, Flinders University
• David Cawthorne, Charles Darwin University

Presenter bios

Presenter abstracts

Hayden Croft

Head of Sport, Exercise and Health, Otago Polytechnic, New Zealand

Hayden Croft is currently the Performance Analyst for the Silver Ferns netball team and was part of the team that won the 2019 Netball World Cup. Subsequently, they went on to win the ISPS Handa Team of the Year Award and the Halberg Award (supreme). They also won New Zealand’s Favourite Sporting Moment.

Hayden is the Head of Sport, Exercise and Health at Otago Polytechnic, and is also the Academic Leader for the Performance Analysis postgraduate programme. Hayden’s expertise lies in a variety of areas all focused on advancing coaching using innovative technologies and performance analysis. This has included work with the Silver Ferns, the All Blacks, the Highlanders, Otago Rugby, World Rugby, and Southern Steel. He has a Master of Physical Education (Biomechanics) and a Bachelor of Physical Education (Sports Science Prescription), both from the University of Otago. Hayden was also the winner of the Technical Support (2020) and Innovation in Sport (2015) categories at the Otago Sports Awards. He is currently working on his PhD (Performance Analysis) at Auckland University of Technology, which focuses on guiding coaches through large data sets. He also holds the Chairperson position with the Oceania branch of the International Society of Performance Analysis in Sport. Hayden provides an immersion style of learning for undergraduate and postgraduate students.

How performance analysis helps not only the Silver Ferns, but also Otago Polytechnic

Performance analysis is a growing field within elite sport and is valued by many coaches as an important part of the coaching process. The performance analyst provides evidence through video feedback and statistics to help players improve, and for the coaches to assess the effectiveness of game plans and strategies. Silver Ferns has been at the forefront of this and has implemented innovative approaches to measuring performance and enhancing player learning. During the build-up to the 2019 Netball World Cup, the Silver Ferns carefully progressed player learning and understanding of a strategy that incorporated the ‘New Zealand style’ of play through their performance analysis systems and coaching. This could have been one of the many contributing factors to their success and ongoing improvements in the game. During this World Cup, Otago Polytechnic students supported the Silver Ferns through remote analysis of the opposition, utilising television broadcasts to capture footage and create match statistics in scout reports. This allowed students to learn how performance analysis works in a real-world setting, and to understand the pressures and expectations of the role. Many of these students have progressed onto careers as performance analysts with employers such as Southern Steel, Northern Stars, Otago Rugby Football Union, and Hawkes Bay Rugby Union.

Myles Cronin

Customer Success Manager, Explorance

Myles Cronin joined Explorance in 2021 as a Customer Success Manager after 7 years of corporate experience in service delivery and business process management, and is focused on product adoption strategies designed to drive business results. Myles has led global service delivery teams in various industries to deliver programs for onboarding, talent development, learning analytics, performance management, and service delivery.

At Explorance, Myles partners with clients to execute tailored technical configurations, as well as develop product adoption strategies. In his role, he is responsible for ensuring our institution partners derive the optimal benefit from their investment. With a focus on creating an effortless experience, Myles ensures the satisfaction and success of our partners on a day-to-day basis, creating strong advocates within our partner organisations.

Optimising response rates in a time of uncertainty: learnings from the HE community

Universities have demonstrated an eagerness to maintain student wellbeing and engagement during the pandemic period. At a time when student feedback has been crucial to informing institutional responses to remote learning, Explorance has sought to examine how to optimise response rates during times of uncertainty and to share the findings and key insights from within the higher education community.

As we know institutional culture, faculty leadership, and ease-of-access are key factors to response rate optimisation, however, mode of course delivery has paved the way for a shift in student engagement. Findings indicate that response rates vary by mode of administration. Moreover, predictors of response differed by administration group. Results are discussed considering the recent surge of administration in online learning due to COVID-19. This is a reflection on the effective strategies and methods applied to administer the student experience across a range of Australian universities.

Stephanie Dykes

Executive Director of Institutional Effectiveness, North Seattle College, USA

Dr Stephanie Dykes is an out and proud transwoman, living in Seattle, Washington, in the United States. She works as the executive director of institutional effectiveness at North Seattle College, and has over 30 years’ work experience in education and business.

She earned her PhD in educational research and measurement and her Master of Arts in Teaching from the University of South Carolina in the United States. She completed her undergraduate work in history and business administration from Clemson University, which is also located in South Carolina.

Dr Dykes has spoken at numerous conferences in the United States and Canada on the topics of gender and women’s studies, as well as institutional research. She has spoken at colleges and universities across the United States about her experiences as a transgender woman.

Dr Dykes is the father of a 28-year-old son, and she sings bass with the Seattle Men’s Chorus. Dr Dykes is also a member and featured soloist at Holy Spirit Lutheran Church in Kirkland, Washington.

Transgender in the academy: challenges and opportunities

This presentation will bring together data sources on the experiences of transgender and gender-diverse students and colleagues from a wide variety of sources. By sharing this information at the AAIR Forum, attendees at this presentation will be better able to understand the experiences of their transgender students and colleagues. This will help institutions to be better able to help their transgender students achieve their educational and career goals.

The information shared in this session is both timely and relevant. In recent years, transgender and gender-diverse persons have gained greater visibility and acceptance. This has been experienced in all aspects of our society, including higher education. Despite gains made in recent years by the transgender community, barriers remain. Increasingly transgender and gender diverse students are enrolled in our institutions. Transgender persons work in every field of human endeavour, including higher education. This presentation will provide attendees with the opportunity to gain knowledge about transgender persons from an institutional research professional who is herself, transgender.

Rob Dalitz

Senior Information Analyst (Institutional Research and Reporting) and Survey Manager, University of Canberra

Rob Dalitz is the University of Canberra’s survey manager. In the past, Rob has worked for Universities Australia (UA), where he produced the first UA data sharing agreement, and performed an analysis of sectoral higher education data for UA. Rob has worked in higher education as an academic and professional, in the private sector, and in government in several analytic and project management roles.

An audit of student surveys across the Australian higher education sector

This presentation reports the results of a survey of how Australian Network of University Planners (ANUP) member institutions manage student surveys. There were 16 respondents, with one respondent typically not providing useable free-text responses. Topics covered included:

• course/program surveys
• unit surveys
• linkage between unit and course/program surveys
• other student surveys
• survey software/system used
• incentives, engagement, privacy, and analysis of student survey data
• analysis of free text in surveys
• reporting of survey results
• usage of survey data
• closing the loop.

Greg Jakob

AAIR Treasurer

Greg is now a higher education consultant, having worked in tertiary education for almost 15 years in a variety of roles including planning, student administration, quality assurance, risk management and governance, campus management and student experience. He has also led and been engaged in several sector-wide and government groups and projects involving student demand management and service pricing; quality systems, policies, and processes; technology development and innovation; and service design and continuous improvement. Greg is highly regarded in the tertiary sector and across government, and his extensive experience is well placed to assist clients with:

• strategic and operational planning
• demand analysis
• financial modelling and forecasting
• business case development
• technology implementation and development projects
• change management.

Kathie Rabel

AAIR Immediate Past President

Kathie is the Immediate Past President of AAIR and has been a life member since November 2019. She was President from 2016–2018, Vice-President from 2012–2016, and was an Ordinary Member of the Committee from 2010–2012. She also served as Acting President of AAIR from 2018–2020. Kathie worked in universities for most of her working life, starting in the US where she was born, followed by some stints in universities in Christchurch and Dunedin, and finishing up in Wellington, New Zealand in 2019. Now retired, Kathie keeps busy by volunteering for AAIR and other organisations, helping where she can.

The case for boosting investment in BI

‘Without big data, you are blind and deaf and in the middle of a freeway.’

– Geoffrey Moore

University business models are being disrupted at an ever-increasing rate, be it through shifting demand for higher education, technological changes to learning, teaching and administration, or by external events such as pandemics and globalisation. Now, more than ever, they need to invest in data as an asset to extract insights and business intelligence to inform strategic and operational decision-making, and to monitor performance. This presentation will outline the business case for boosting the investment in BI as an asset management strategy rather than a cost of doing business.

Lisa Bolton

Director, QILT Research and Strategy, Social Research Centre

Lisa has been working in the QILT program at the Social Research Centre since January 2015. Prior to this, she spent 29 years in the VET and university sectors where she has extensive experience in a range of roles—including teaching, curriculum review, academic management, quality management, strategic planning, organisational research, surveys, and evaluation. She has a strong interest in institutional research, as well as strategic planning, quality, and evaluation in the tertiary education sector.

Lisa has a Bachelor of Education from Deakin University (Rusden), CIV in Vocational Training and Assessment from Swinburne University, and a Master’s degree in Education (Leadership and Management) from RMIT.

Combatting survey denial

Surveys and evaluation SIG

As the moon turns, so too do those who believe that student feedback surveys are the work of the devil to be rooted out at all costs. What arguments can be made to combat the criticisms that seem to return year after year?

In this SIG session, we will discuss some of the main criticisms levelled against student feedback surveys and discuss ways in which these might be mitigated so that survey data can be used effectively to improve the student experience.

Andrew Bradshaw

Manager, Statutory Reporting, Macquarie University

Andrew Bradshaw is the Manager of Statutory Reporting at Macquarie University and has been at the university for 31 years. Over this time, Andrew has seen many changes implemented by the Department on government funding and statutory reporting. He has an expansive knowledge of government requirements, in both load management and planning, and in government reporting.

COVID-19 and the impact on university load planning

Load management and planning SIG

With some states opening their borders for international travel, what effects will COVID-19 continue to bring in 2022 and future years?

This session will provide a platform for universities to share their intended approach and the assumptions they are making in load planning for 2022, especially in relation to SLE and student progression. Unfortunately, due to time, we will not be able to have a representative from the Department at this session.

Svetlana Shenderova

Researcher, Tampere University, Finland

Dr Svetlana Shenderova is a researcher, expert, and consultant in higher education policies, university governance, and degree programme management. She has worked in nine Russian universities as a teacher and researcher in both economics and higher education, as well as performing administrative duties, and consulting. Svetlana has also investigated the EU–Russian joint programmes as a Key Expert II for the EU Delegation to Russia.

Currently, Svetlana is a researcher of the EDUneighbours project that is affiliated with Tampere University, Finland. She studies institutional environments of internationalisation, for example, Finnish–Russian double degrees. The regional scope of her expertise comprises Russia and post-Soviet countries, including the EU (particularly Finland and Poland), Israel, and China. Svetlana Shenderova holds a Dr Econ. Sc. (2012, St Petersburg State University of Economics), and a PhD in Political Economy (1997, St Petersburg State University, Russia).

Permanent uncertainty as normality: how EU and Russian universities cooperate under the pressure of the COVID-19 pandemic

The pandemic drastically changed the paths, opportunities, and the spectrum of internationalisation activities that had already been implemented, notwithstanding the political tensions. In my presentation, I will address the evolution of practices and approaches to internationalisation activities between the universities of the EU member states, in particular Finland and Russia. In addition, I will focus on the pandemic challenges faced by academics, university administrators, and students involved in internationalisation activities, and the managerial solutions suggested by the partner universities. The presentation is based on the results of the Finnish–Russian research project, ‘Towards good neighbourliness with higher education cooperation’ (EDUneighbours), which studied Finnish–Russian cross-border higher education cooperation and institutional environments inside and outside the universities that implemented double degree programmes between 2014 and 2020.

Chris Norton

Senior Advisor (for Surveys), Wellington Institute of Technology and Whitireia Community Polytechnic, New Zealand

Chris has worked in the polytechnic sector in New Zealand and the Middle East over many years. He started as a part-time tutor in computer applications, then moved through programme management, quality assurance, and international initiatives before taking on the role in Wellington Institute of Technology and Whitireia Community Polytechnic as the Senior Advisor responsible for managing the collection, analysis, and reporting of feedback from the student surveys systems.

Surveys Automagic: connecting systems to automate student surveys/evaluations: learnings and outcomes, pressure points, failures, and successes

Having used Watermark’s Course Evaluations & Surveys (formerly EvaluationKIT) since 2017, we embarked on developing methods to auto-populate surveys with courses and students via the API.

In this presentation, I will outline the supporting documents and processes we used to make this process viable—as well as reveal the new pressure points—as we seek to ramp up improvements in student and staff engagement.

Peter Hopwood

Higher Education Practice Lead, Altis Consulting

Peter has specialised for the past 25 years in data and analytics, with a proven track record in delivering and directing teams to deliver large systems to multinational companies in the UK, Europe, and Australia. Peter also set up and managed Altis’s UK business for 7 years.

Peter is now the Higher Education Practice Lead for Altis and has worked with over a dozen universities in the UK and Australia. Peter is currently based in Sydney.

Peter has spoken at many conferences, including: The Data Warehousing Institute (TDWI) conference in London, the IBM Business Analytics Forum, and the Australasian Association of Institutional Research (AAIR) Forum in Australia. Peter has been published in DM Review in the USA, CIO Magazine, IT for CEOs and CFOs in the UK. He has also been interviewed in MIS Magazine in Australia, CIO Magazine, and Computing in the UK.

Harpreet Khalsa

Chief Data Officer, Edith Cowan University

Harpreet is a firm believer in information as a ‘real asset’, and if utilised prudently with well-understood business objectives, it can make wonders happen. Working with business leaders in both consulting and industry has helped Harpreet acquire business acumen, and that combined with his passion for converting data into actionable insights, has helped him deliver on a variety of meaningful analytical and forecasting capabilities.

The focal point of his professional journey has been a never-ending fascination with practically demonstrating how technology can be exploited to elevate the quality of decision-making and functions performed by human capital.

Currently, Harpreet is working as a Chief Data Officer for Edith Cowan University, and is responsible for management reporting, analytics and data quality, and governance for the University. He has been pivotal in creating the culture of insights-driven decision-making, which has led to significant improvements in planning and operations within various parts of the organisation.

A, B, C, 1, 2, 3 – Replacing ECU’s data platform

Edith Cowan University (ECU) leadership is increasingly asking for more data, from more systems, combined and curated in ever more sophisticated ways.

ECU had a data warehouse that had been developed on-premises over 10 years ago, which housed the crucial data needed but was slow to develop, expensive to maintain, and cumbersome to operate.

The University decided to move the warehouse to the cloud utilising Microsoft Azure and Databricks. The new design was metadata-driven, meaning the data transformations were was no longer lovingly hand-crafted, but configurable to rapidly ingest and transform new datasets. This metadata-driven design also used patterns—1, 2, 3 and A, B, C—to streamline the process and minimise staff time for maintenance and development.

This presentation will outline ECU’s journey replacing the data and analytics platform over the last year.

Julia Petrou

Senior Surveys Coordinator, La Trobe University

Julia Petrou has over 23 years’ experience working in the tertiary education sector at La Trobe University. She holds a Bachelor of Science and enjoys working on strategic projects that involve data-driven evidence and business improvement processes to add value to the changing organisation. In her current role leading the surveys team at La Trobe University, she is responsible for the operations and reporting of core University surveys, as well as monitoring and reviewing survey-related systems, processes, and internal policies.

Hailey Bynon

Survey Analyst, La Trobe University

Hailey Bynon is new to the higher education sector, having commenced in the surveys team in March 2021 as a Survey Analyst at La Trobe University. She administers and analyses staff and student surveys, modelling the data into reports and dashboards to identify trends that help inform the direction of the university. Prior to her time at La Trobe, Hailey completed a Bachelor of Business and spent 5 years working in the events sector where she analysed the economic impact of events, and this is where she established a passion for data. She seeks to share her passion and unlock the power of data to drive change and innovation.

Work-based learning: a central framework for student and host feedback

A fundamental requirement of the Higher Education Standards Framework (Threshold Standards) is for university providers to quality-assure all work-based learning activities. A centralised evaluation approach for both students and host organisations has been developed to enable feedback through a formal survey mechanism. This system required a collaborative project that examined the requirements across the university. In this presentation, we will share the challenges faced in developing a single evaluation mechanism, data preparation, and the reporting outcomes used to shape La Trobe’s student support programs.

Tony Nolan

Data Mathematician and CEO, G3N1U5 Pty Ltd

Anthony (Tony) Nolan OAM was an AAIR committee member from 1997 to 2006, and first presented at an AAIR forum in 1996. Tony was the Information & Intelligence Officer at UTS from 1995 to 2003. After leaving UTS, Tony went to the ATO and worked as an intelligence officer, data scientist, and finished as the innovation officer in the data science specialist profiling unit. Tony then worked with the predictive crime team in the NSW Police Force where he headed the data science section on advanced profiling and predictive techniques. Tony is currently retired and is developing new advanced cohort profiling in his own company. He is looking forward to not being retired too much longer and is seeking new challenges.

Applying capstone profiling to student cohorts

Since leaving UTS in 2003, I have been developing profiling for my roles in the ATO and the NSW Police Force. As an intelligence officer and data mathematician, I have seen new ways to use traditional methods in combination with open-source data and open-source software, to generate enhanced datasets for cohort profiling. This presentation will demonstrate some of the methods behind creating enhanced datasets, using AI and machine learning, and using the advanced features of spreadsheets for intelligence gathering and decision support.

Sonia Whiteley

Managing Director, Maven & Edge

Sonia Whiteley has worked as a mixed-methods, applied social policy researcher and program evaluator for more than 30 years. She specialises in educational research, with a particular interest in graduate outcomes, student retention, social and emotional development of young people, and school readiness.

Sonia is currently the Managing Director of Maven & Edge, an independent strategic research consultancy that supports the government and institutions to understand the hopes, dreams, and ambitions of potential students, current students, and past students in the context of an ever-changing educational landscape. Recent projects have focused on exploring the nature of the ‘new’ student experience and reviewing survey materials to assess the extent to which they are still fit for purpose. In her ‘spare time’ she enjoys walks on the beach, reading about Victorian-era palaeontology, and running a couple of gyms.

Freedom of expression & the experience of students at Australian higher education institutions: development of a survey module

In late August 2019, the Minister for Education, Dan Tehan, announced that the Student Experience Survey (SES)—one suite of surveys included in the Quality Indicators for Learning and Teaching (QILT) survey program—would be updated to include a module of questions about freedom of expression on campus, on the basis that ‘universities, students and the public should know if students and staff are afraid to raise certain topics’.

Freedom of expression is a critical but challenging concept to define, operationalise, and measure. Qualitative and quantitative data were collected and analysed to support the effective and efficient design of a targeted set of items that are consistent with those used in the QILT survey program.

This presentation summarises the development and design of the freedom of expression module that was included in the 2021 SES.

Dick Audley

AAIR member

Dick Audley has been involved in the assessment of higher education programs for much of his professional career, where he pioneered many of the techniques now used for the analysis of courses. After retirement, Dick completed a PhD at UTS where he examined the impact of risk management on the course accreditation procedures of Australian universities. The transfer from assessor to student involved the development of a different type of relationship with a range of institutions, almost all of which had strict protocols about interviewing staff and the conduct of research into their activities.

The conduct of research into Australian universities

The inner workings of Australian universities have tended to be off limits to researchers, possibly because few are willing to risk exploring the hidden aspects of potential employers, and some institutions seem comfortable having their internal procedures made public for inspection or discussion. This means that there are limited precedents to guide those who undertake doctoral research in this field. At the same time, there is no reason why the procedures used to examine the internal workings of corporate, professional, or charitable bodies should not be applied to tertiary institutions. The question is how might this best be done, and how can one maintain the goodwill of those being investigated while doing so?

The presenter will outline the number of difficulties he encountered while conducting research into the procedures of Australian universities, and he will explain how these were successfully overcome. Universities encourage research, but rarely into themselves, and it is hoped that future scholars may benefit from his experience.

Dr Bekki Green

Learning Analytics Coordinator, CQUniversity

Dr Bekki Green joined CQUniversity in 2020 as the Learning Analytics Coordinator, where she immediately began plotting to use QILT data to answer the University’s big questions. Bekki has worked in research evaluation and online education, and she has a particular focus on learning and teaching research. With a PhD in Applied Psychology, Bekki takes a psychological perspective on addressing student motivation and teaching strategies for developing graduate outcomes. You will find her 6-coffees deep in data analysis or on LinkedIn, if you’re looking.

Unpacking learner engagement: an analysis of QILT data

The Quality Indicators of Learning and Teaching (QILT) Student Experience Survey (SES) contains six focus areas that are used for benchmarking and strategic improvement planning. Learner Engagement is a focus area that has received consistently lower ratings across the sector, with the 2019 national average indicating only 60% of students were satisfied with this area compared to 80% of students satisfied with teaching quality. In 2020, the COVID-19 pandemic showed a disproportional impact on learner engagement, with satisfaction dropping from 60% to 43%. Universities with large cohorts of online students also show poorer performance on learner engagement. The impact of online cohorts and COVID-19 on learner engagement has been attributed to the online study mode and the suitability of learner engagement survey questions for assessing an online study experience.

This presentation will unpack the factor structure of the QILT SES focus areas, discuss predictors of learner engagement, and provide recommendations for the sector in interpreting and improving learner engagement scores.

Charles Palmer

Solutions Architect, Australian National University

Charles is a Vietnam veteran who disappeared into technology and coding, games, and simulation development up until 1981 whilst an undergraduate at CCAE. He is a self-taught assembler, comms developer (submarine comms), and Navy refit management software developer. Charles was a systems auditor with PWC, and has also worked as a freelance data migrator, payroll review, and migration expert, as well as being a part- and full-time academic teaching information security and management. Charles published his master’s thesis, An approach for managing data quality, in 2012 through the University of Canberra Library. More recently, Charles has worked as the Chief Technology Officer at ACT Health, was the Deputy Manager ICT Security at ActewAGL, Enterprise Architect Catholic Education diocese, and is now the Solutions Architect at ANU. He has commenced a PhD focusing on data and information assurance in an enterprise.

Data and information assurance in an enterprise

Organisations manage voluminous information that is expected to support their decisions, so the problem of poor and variable information quality becomes, in many ways, costly. Why then do many organisations fail to manage their information and quality issues effectively; or even at all?

This presentation proposes a framework that addresses quality measures in a practice-management tool that continuously adapts to each organisation and its needs.

Lester DSouza

National Manager, Business Intelligence, Australian Catholic University

Lester has worked in the planning office at ACU for over 13 years. He is currently the National Manager, Business Intelligence, focusing on BI and predictive analytics, but he has also been involved in TAC and direct applications reporting and analysis, as well as enrolment planning and survey governance. Lester is the chair of the AAIR Business Intelligence and Analytics special interest group. When he has hours to burn, Lester spends his time as a video gamer and Sith Lord.

Bye Bye BI

Is this the age of the citizen analyst? More and more businesses are ditching the traditional, tested model of obtaining their reports and analyses via a single unified team. Now employees are encouraged to access, process, and analyse their own data. Be it from a data warehouse, data lake, or from primary data sources. Should Australian universities step up and venture into this uncharted territory, or do we wait and watch? Join us for a discussion on this and other things BI in the HE sector!

Don Johnston

AAIR Vice-President

Don Johnston is the Manager, Reporting and Analysis at Southern Cross University, Australia, as well as the Vice-President of AAIR.

Don has been the AAIR Vice-President from 2016–2018, and was appointed Vice-President again in early 2020. He was an Ordinary Member of the AAIR Executive Committee from 2006–2016 and 2018–2020. Don is also the Special Interest Groups portfolio leader with AAIR, and the AIR contact.

Trying
Complicated
Stressful
I don’t know what to …

How did you survive 2021 with moving to TCSI reporting? Did you survive? What was your experience? How will you be continuing into ‘business as usual’?

Please join us for a TCSI ‘counselling’ session, where we can both share our stories and journeys so far, as well as discuss how we are planning for the future.

Phil Blagus from the Department of Education, Skills and Employment (DESE) will join us for an update on TCSI.