Core area 1: Operational issues

b)     Technical knowledge and ability in the use of learning technology

My earliest memory of using a computer was being praised by my dad for creating a program in basic on my ZX Spectrum when I was about 8 years old.  It was a very simple program that drew a pyramid but I'd figured out how to make every "brick" a random colour and I was incredibly proud of myself.  

Since then I've grown up with computers and have completed a Bachelor's degree at University and also a Diploma of Commerce in Information Systems and a Post Graduate Diploma of Commerce in Information Systems.  I converted my Master's Degree into a PhD but unfortunately did not finish this due to the vagaries of life.  Having said that I've retained my love of using technology to make the world a better place.

One of the projects that I have been involved in was a project to build a Team Based Learning Assignment Marking and Peer Feedback System.

In 2010 The University of Auckland Business School rolled out two new courses (Business 101 and Business 102) that were to be taught using the Team Based Learning method.  Our Dean at the time was keen for everyone in the Business School to have an exposure to this way of teaching and so I put my hand up to be involved.

The BUSINESS 101/102 design team envisioned an innovative assignment that required students to review, not only their own work, but that of six of their peers. The pedagogical goals of the team provided a compelling justification for their desired approach. With a small course, the desired approach could have been done using a paper-based system. However, with around 1,500 students in BUSINESS 101 (and in the second semester, when BUSINESS 102 begins about 2,000 students) a paper-based system was nigh on impossible.

The students are spread across 15 streams. Within each stream, students work in teams of 5–6 members. Each student has to submit two written assignments.

It very quickly became apparent that there was no off the shelf software that could facilitate this.  I somewhat naively volunteered to see what I could do in my spare time.

The BUSINESS 101 design team presented me with the following requirements:

  1. Assignment submission: Each student anonymously submits an electronic copy of their assignment. The assignments are not allowed to have any identifiers on them as both the review and marking process is double-blind.
  2. Individual peer review: Each student then gets 6 of their peers’ assignments to review. They get 2 from the top third of the class, 2 from the middle and 2 from the bottom third. To further complicate matters no student will review an assignment from within their stream and every member of a team will get the same bundle of assignments to review. Each student also has to review their own assignment after reviewing their peers. This means that there are approximately 10,500 (1500x7) assignment peer reviews taking place. The students use a rubric to provide both quantitative and qualitative feedback on each assignment.
  3. Team peer review: Each team of students then meet in class to compare and moderate the feedback each member has provided for each assignment. They then recommend a grade and provide further feedback to the author of the assignment as a team.
  4. Expert marker review: The expert marker gets a copy of the still anonymous assignment, the 6 individual peer reviews and the individual’s self-review as well as the team review. The expert marker reviews the assignment, adds their feedback and allocates a grade.

As you can see, this was not a trivial task.  I built two versions of this system.  One using a combination of Google Forms and Microsoft Office Automation  (v1) and another using PDF Forms and PDF Automation (v2).  I had help from one of our Technical Specialists (Viral Shah) who wrote the algorithm that matched students and reviewers.  You can see the details of how it worked here.

Evidence that this was useful:

In 2011 I received a University of Auckland Business School Teaching Excellence award based on the rollout of this system.  You can see my award evidence portfolio here.

Our Dean at the time wrote the following about it: 

Andrew Eberhard (ISOM) and Viral Shah (ILT) have devised a system that enables the BUSINESS 101 essays to be captured, anonymously peer reviewed and graded using the University Google Documents facility. The system is user friendly and has enabled the assignment review process to work amazingly well even with the massive numbers involved. The innovation has saved countless days of administration, significant paper savings and ultimately made a seemingly impossible exercise plausible, resulting in a seamless student experience.

Reflection:

I was discussing this project with one of the members of the Business 101/102 design team, Dr Peter Smith, and he compared me to a stunt pilot.  A highly skilled and effective pilot who can fly “systems” that ordinary pilots cannot.  The user experience in this project was very good but it had required a huge amount of time from me, behind the scenes, in administering it.

This project made me realise that I needed to stop building custom solutions that required extensive input from a “stunt pilot” and instead, focus on using on the shelf software that “ordinary pilots” can use.   I needed to change from building systems that required me to admin them, to finding solutions that I could hand off to ordinary academics.

I credit the huge amount of work I put into this project with changing my perspective and making me far more useful to my colleagues.