AAIR Forum 2020

Theme

2020 It’s not all Zoom and Gloom: Let AAIR lift you up by sharing our experiences

Subthemes

The 2020 AAIR Forum subthemes include:

• Impacts of COVID-19
• The Evolving Role of the Institutional Researcher
• Justifying the Value of Higher Education and research
• Evidence-based Strategic Planning (and Implementation)
• Benchmarking for Continuous Quality Improvement
• Informing Learning and Teaching through Evaluation and Surveys
• Realising benefits – the role of Institutional Research in Enterprise and Transformation Projects
• Compliance and Government Reporting Reform
• Business Intelligence, Analytics, Visualisation and Modelling – the Foundations of IR

Special interest groups

• Business intelligence and analytics
• Load management and planning
• Government reporting
• Surveys and evaluation

Organising committee

For the virtual forum in 2020, the organising committee is the current AAIR Executive Committee:

• Stuart Terry, Otago Polytechnic (AAIR Forum Chair)
• Don Johnston, Southern Cross University
• Pam Rayner, Bond University
• Kathie Rabel, Life Member
• Greg Jakob, Federation University
• Lisa Bolton, The Social Research Centre
• Andrew Bradshaw, Macquarie University
• Leone Nurbasari, The Australian National University
• Eva Seidel, Flinders University
• Penny Szybiak, Charles Darwin University

Presenter bios

Presenter abstracts

Tim Brennan

General Manager (Asia-Pacific), Explorance

Tim Brennan is the General Manager of Explorance, Asia-Pacific. Based in Melbourne, Australia, Tim is leading the rapid growth of Explorance in this region.

Prior to joining Explorance, Tim was the Senior Manager of the Survey Services Centre at RMIT University, Australia’s largest tertiary institution, where he was responsible for the administration and reporting of student feedback across all global locations.

Tim’s experience extends across government, higher education, and the vocational education sectors, with a background in statistics and strategic planning.

Preparing for the new normal: How can effective student feedback help universities to survive and thrive in the next academic year?

The unprecedented challenges brought about by the COVID-19 pandemic has led to major disruption for the higher education sector worldwide.

The initial shift to online teaching from March 2020 raised serious questions around how the student voice can be captured effectively (through unit/teaching evaluation surveys
and other student feedback mechanisms), especially given the sector’s reliance on traditional approaches.

Beyond the quality assurance rationale and irrespective of the mode of delivery, ensuring that current students are generally happy with their institution’s response during this difficult period, and specifically that their feedback around modules/courses/campus life and facilities are listened to, is paramount. It is not only a student retention consideration but is also of importance in minimising the threat of those students potentially posting negative reviews or comments on influential online forums that could hamper future student recruitment activity.

How feedback from students about their teaching and learning experience and wider student experience is captured is critical to institutional performance and longer-term strategy. Making available insights to inform successful approaches and devise timely interventions where necessary cannot be overstated.

In this session, Explorance brings together a meta-analysis of how universities across the globe are engaging the student voice in our ‘new normal’.

Joshua Lee

Manager, Institutional Analytics and Data Science, The University of Sydney

Joshua Lee is Manager of Institutional Analytics and Data Science at The University of Sydney and a former lecturer in the Discipline of Business Law.

He was once human, but 2020 – like so many other things – changed all that. Now he is a man-machine-data hybrid, charged every waking hour with solving the great mysteries of our time. Where are our students? Who have they been in contact with? How many will return if classes move online?

Diamonds are made under pressure: inside the University of Sydney’s data-driven response to COVID-19

In this rant exhilarating presentation, find out about how the University of Sydney harnessed the power of Snowflake and Power BI to answer COVID-19 reporting challenges across the entire gamut of university operations – from finance to planning; administration to teaching; research to space management; HR to cybersecurity – and how the lessons learnt and advances made are being used to build the BI and analytics platform of the future.

Eric Wang

Integration and Performance Reporting Lead, Monash University

Eric is an integration and reporting lead in the University Planning and Statistics department at Monash University. He is responsible for delivering end-to-end data integration and reporting solutions to enable evidence-based decision making that drives load management and planning activities.

Prior to his current position, Eric has worked in both education and international offices as an analyst providing evidence to support the university’s strategy development and major initiatives.

Uplift QILT data utilisation using Excel Online and Power BI

The QILT data enables the university to understand our performance in key areas across a students’ lifecycle from commencement to employment. The data allows us to benchmark our performance with other institutions, thereby underpinning opportunities and identifying areas needing improvement. However, the data are complex and thus it requires significant technical effort and collective domain knowledge to fully unlock its insights.

This presentation introduces a reporting solution based on off-the-shelf Excel Online and Power BI. Users have access to centrally curated, and readily available, QILT data via a user-friendly interface. It is flexible and easy to use and inherits benefits offered by the Power BI server, such as low cost, access control, and scalability. More importantly, it brings federated governance to our QILT data. It allows subject matter experts (SMEs) across the university to develop, share, and maintain their Power BI reports based on the same data that underpins our reporting solution.

Since its release, more and more SMEs are developing their narratives using this approach. We’ve developed closer working relationships and seen more frequent and constructive engagement with faculty staff. All these improvements are resulting in increased utilisation of our QILT data.

Lisa Bolton

Director, QILT Research and Strategy, Social Research Centre

Lisa has been working in the QILT program at the Social Research Centre since January 2015. Prior to this, she spent 29 years in the VET and university sectors where she has extensive experience in a range of roles—including teaching, curriculum review, academic management, quality management, strategic planning, organisational research, surveys and evaluation—and has a strong interest in institutional research, as well as strategic planning, quality, and evaluation, in the tertiary education sector.

Lisa has a Bachelor of Education from Deakin University (Rusden), CIV in Vocational Training and Assessment from Swinburne University, and a Master’s degree in Education (Leadership and Management) from RMIT.

Deep dive into graduate outcomes: more arts than science

Surveys and evaluation SIG

In the context of the current higher education policy debate around ‘job-ready graduates’, understanding and utilising the information available through the QILT Graduate Outcomes Survey (GOS) has become ever more vital for Australian institutions.

One such issue that has risen to prominence is discussion around the labour force outcomes of recent graduates in science and mathematics versus humanities courses. Just looking at high-level indicators in national reports hides a wealth of nuance around study area breakdowns and definitions as well as further study outcomes for undergraduates, previous attachment to the labour force, and various definitions of underemployment, including the relevance of work to the course undertaken, usual and actual working hours, remuneration, skills utilisation and reasons for underemployment, as well as labour market conditions related to the actual location of the graduate rather than the main campus of the institution.

This session will explore some of these more in-depth concepts using recent and medium-term graduate outcomes collected through the GOS and GOS-L.

Andrew Bradshaw

Manager, Statutory Reporting, Macquarie University

Andrew Bradshaw is the Manager of Statutory Reporting at Macquarie University and has been at the university for 31 years. Over this time, Andrew has seen many changes implemented by the Department on government funding and statutory reporting. He has an expansive knowledge of government requirements, in both load management and planning, and in government reporting.

Job-ready graduates package

Load management and planning SIG

In the Load Management and Planning SIG, Andrew Bradshaw will introduce our guest, Andrew Herd, from the Department of Education, Skills and Employment.

Andrew Herd will share the current news on the Job-ready graduates package, bidding for national priority and short courses, plus new funding agreements for 2021. Members present will be able to ask Andrew Herd questions during the webinar, and he can either answer these at the time or take them back for follow-up within the Department.

Colin Smithies

Business Intelligence Analyst (Tertiary Performance), Victoria University of Wellington

Colin Smithies is the business intelligence analyst (tertiary performance) for Te Puna Tātari Kōrero—Planning and Management Information (PaMI) at Te Herenga Waka—Victoria University of Wellington, a role he has held for almost 5 years now.

Colin is a microeconomist/econometrician who specialises in building both forecast models for EFTS and prediction models for student performance.

Colin’s responsibilities include, forecasting demand, modelling student performance, implementing MCDA models to determine student choice criteria etc., modelling likely uptake (or not) of new courses/qualifications, calculating market shares in the NZ market, and anything else that requires an economic model builder.

Retention through success: 1000minds (using no major drama) and analytics for student success and early intervention

On average, around 20% of New Zealand first-year university students will attrit in their first year; 50% of those that attrit will have failed all or some of their courses. The cost of this failure can be measured in both dollars (lost revenue and cost to the student in both fees and opportunity) and human misery. Our aim is to act early and to go hard; just like New Zealand did with COVID-19.

As with the COVID-19 strategy, identification is key; to enable early and efficient action requires targeted interventions.

First-year students lack full information on two counts, what degree they want to do in the long run, and that they are prepared adequately enough to attempt that degree. By using NO MAJOR DRAMA, and by modelling likely success, we can help students make better choices by providing a more complete information set that combines both what they want to do and that in which they are adequately prepared to achieve.

When combined with continued monitoring of engagement with the course(s) and using well-timed interventions when student behaviour/achievement is highlighted as less than optimal, then the literature says that attrition rates will fall. This is not just a financial imperative for universities but a moral imperative as we need to minimise human misery by promoting success across the student body.

In this presentation I will detail the model we use to help identify students that could benefit from targeted interventions and talk about the road to arriving at this model amongst the myriad of alternatives available. I will also discuss (and demonstrate) how this model sacrifices some accuracy to minimise harm, i.e. to make sure no one is left behind.

This will not be a technical diatribe, but a worked example of how to build the dataset and implement a similar model.

Rintaro Ohno

Senior Assistant Professor, Tohoku University

Rintaro Ohno has studied physics and mathematics at Würzburg University in Germany, received his PhD in Information Sciences at Tohoku University in Japan, and is currently the Senior Assistant Professor in the Strategic Planning Office at Tohoku University, Japan.

He specialises in complex analysis and geometric function theory and is currently teaching linear algebra. He has also taught English and German for freshman classes and provides a wide, interdisciplinary perspective on institutional research and related projects.

Student experience in Japan

The year 2020 has certainly been a difficult and challenging year. Everybody all over the world has had to adjust their lives to the current events, and the rapid changes have left many in discomfort and worries about the future. The situation on campus is definitely not different, since students and teachers were forced to adapt to rapidly changing rules and circumstances as well.

This presentation will briefly talk about current events in Japan, especially at Tohoku University, and focus on the outcome of the survey regarding the shift to online classes. In particular, we will discuss the different opinions of freshmen students, who were supposed to start their new campus life on 1 April 2020. We will also discuss sophomores and above, who had already established social connections and relationships on campus, and teachers who had to change their scheduled program and adapt to online classes and remote teaching.

Winner: Best presenter award

Anthony Claridge

CIO, Study Group

Anthony has worked globally across many industries, with business leadership positions in the medical sector, with ResMed, where he was in charge of global supply operations, and more recently as the Chief Operating Officer of ResMed’s German healthcare business. He was also the scientific sector at Leica, where he was responsible for global processes.

Anthony has held various technology roles, including CIO within the life science, industrial and software sectors, with companies including United Technologies and Xerox, where he worked closely with business leaders to streamline processes and systems architecture to speed up internal processes and make life easier for their customers. Anthony has worked in the USA, Germany, France, the United Kingdom, and Australia.

Peter Hopwood

Higher Education Practice Lead, Altis Consulting

Peter has specialised for the past 24 years in data and analytics, with a proven track record in delivering and directing teams to deliver large systems to multinational companies in the UK, Europe, and Australia. Peter also set up and managed Altis’s UK business for 7 years.

Peter is now the Higher Education Practice Lead for Altis and has worked with over a dozen universities in the UK and Australia. Peter is currently based in Sydney.

Peter has spoken at many conferences, including: The Data Warehousing Institute (TDWI) conference in London, the IBM Business Analytics Forum, and the Australasian Association of Institutional Research (AAIR) Forum in Australia. Peter has been published in DM Review in the USA, CIO Magazine, IT for CEOs and CFOs in the UK. He has also been interviewed in MIS Magazine in Australia, CIO Magazine, and Computing in the UK.

4 POCs in 4 weeks: cloud platforms and new ways of thinking

Where in 4 weeks, Altis worked with Study Group to identify opportunities for a step-change in the use of data and analytics at Study Group.

We used different ways of thinking about data to investigate the top questions raised by the executive and worked with the Study Group IT team to develop 4 different types of proof of concept (POC) using cloud platforms, exploring answers to these questions.

Eva Seidel

Manager, Load Planning and Data Science, Flinders University

Eva Seidel works at Flinders University in the Planning and Analytical Services team and has been there for almost 10 years. Prior to her current appointment, she worked at the Australian Bureau of Statistics. Eva’s background is in statistics and she has always worked with data. She loves the sector and is especially passionate about building and fostering the great AAIR network.

Load planning at Flinders: our new way of approaching this collaboratively, visually, and continuously

In my presentation, I will talk through how we have transformed the load planning process into one that is more collaborative, visual, and continuous.

During the session, I will share our experiences with introducing the use of PowerBi, Microsoft Teams, and how we take a partnership approach throughout the process to achieve the best outcomes for Flinders University.

Lisa Bolton

Director, QILT Research and Strategy, Social Research Centre

Lisa has been working in the QILT program at the Social Research Centre since January 2015. Prior to this, she spent 29 years in the VET and university sectors where she has extensive experience in a range of roles—including teaching, curriculum review, academic management, quality management, strategic planning, organisational research, surveys and evaluation—and has a strong interest in institutional research, as well as strategic planning, quality, and evaluation, in the tertiary education sector.

Lisa has a Bachelor of Education from Deakin University (Rusden), CIV in Vocational Training and Assessment from Swinburne University, and a Master’s degree in Education (Leadership and Management) from RMIT.

The impact of COVID-19 on the student experience

There have been many articles about ‘disruptors’ coming down the pipeline that would affect higher education in Australia, with online delivery looming large amongst them. Some higher education institutions were well down a path of innovative, well-designed, fully online, or mixed-mode programs, often alongside more traditional on-campus delivery, with an ever-growing faculty familiar with online student engagement. However, the sudden lockdowns associated with COVID-19 forced this transition on many unprepared institutions, in study areas that relied heavily on specialist laboratories and equipment or work-integrated learning, and on a sometimes unwilling and unready student population.

In this session, we will discuss the way in which QILT Student Experience Survey data can be utilised to assist in measuring the level of disruption and recovery on aspects of the 2020 student experience due to institutional responses to COVID-19 and what other data we might need to ‘tell the story’ in our institutions.

Lester D’Souza

National Manager of Business Intelligence, Australian Catholic University

Lester has worked in Australian higher education for over 12 years, all of them in the planning office at the Australian Catholic University. He has worked as a data analyst focusing on TAC and direct applications, enrolment planning, etc. More recently, Lester has taken the position of National Manager of Business Intelligence. Lester is also the current chair of the AAIR Business Intelligence and Analytics SIG.

The next logical step

It is predicted that by 2022, most organisations with mature IT infrastructure will leverage machine learning-enabled data quality technology for suggestions to reduce manual tasks for data quality improvements.

That sounds far-fetched! Or does it?

So what can we actually expect and what can we realistically do?

Michael Rothery

Senior Manager, Planning and Performance Analysis, University of Technology Sydney

Michael Rothery is senior manager, planning and performance analysis in the Planning and Quality Unit at the University of Technology Sydney. The scope of the role includes oversight of student load planning, statutory reporting, corporate planning, and enterprise performance reporting.

UTS student load plan and intake plan

At UTS, the student load planning model provides a projection of the university’s student load, headcount, and load-based income across 500+ courses for the coming year and beyond. An integrated intake plan provides leading indicators of conversion rates at course admission to monitor progress against commencing load targets.

The new solution has been built on a cloud-based IBM Planning Analytics platform powered by TM1®, which has replaced old Excel student load models. It supports real-time collaboration across faculties and units. This visibility supports better annual budgets, planning, long-term financial plans, and setting faculty targets and admission turnstile quotas. Ultimately, this helps UTS meet its strategic goal of using data to support planning processes and guide decisions about the future of learning.

Dr Ben Phillips

Chief Survey Methodologist, Social Research Centre

Ben is a survey researcher and methodologist with over 17 years of experience in social research, primarily in the US, working for academic research institutes and government contractors.

He has extensive experience with all phases of the survey lifecycle from writing proposals, questionnaire development and cognitive testing, sample design, managing data collection via in-person, paper, telephone and web modes, through to data processing, quality control, weighting, analysis, reporting, deidentification and data archiving. Ben’s research interests include web surveys, developing efficient sampling schemes for studies of rare ethnic and religious populations, and optimal sample allocation.

Ben holds a BA with first-class honours jointly in government and public administration and in Jewish civilisation, thought and culture from the University of Sydney. He also holds an MA and PhD from Brandeis University, jointly in sociology and near eastern and Judaic studies. He is a member of The Research Society and the American Association for Public Opinion Research, where he serves on the Standard Definitions committee.

Email experiments in QILT: using evidence to improve online survey response

Email is the primary mode of delivery of survey invitations and reminders in the QILT studies. In the interest of maximising survey completion, the Social Research Centre has systematically experimented with subject lines, text in the email body, imagery, and day of sending. We will present the results of email experiments undertaken from 2018 to 2020, and we will draw on lessons learned from communications with the student and alumni populations.