Monday, December 14, 2009

Teaching for excellence: Excellence in teaching


Earlier this year at the eFest meets Teaching and Learning Conference (hosted at UCOL), I was privileged to take part in the conference wrap up led by Lisa Emerson (Ako Aotearoa) - "Teaching for excellence: Excellence in teaching".

Lisa began by discussing the quote: “The classroom, with all its limitations, remains a location of possibility" (Bell Hooks, 1994, p. 207), and then went on by looking at the notion of excellence in tertiary teaching, referring back to the main theme of the conference. She asked how we can bring student evaluation of teaching into our teaching institutions, and teaching practice. We have clear empirical methods to research outputs; the push for this empirical data then moves to the evaluation of tertiary teaching. How do we measure excellence in teaching. Lisa listed 1) retention, 2) success rates, 3) student evaluation, 4) compliance measures, 5) willingness to engage with new technologies, 6) willingness to show commitment to teaching by engaging in PD, and 6) by willingness to disseminate good practice (e.g. by publishing in academic journals). She then went on to discuss who wants to measure excellent, including the teacher (especially for career development), the institution, political bodies (to measure value for money), and others such as Ako Aotearoa (for example there may be a desire to raise the profile of teaching within the tertiary sector).

Emerson reflected on some experiences she had working with award winning teachers, who were incredibly humble about their teaching practice. She then went on to unpack the following: “It is crucial that we challenge any feeling of shame or embarrassment that teachers who do their job well….For when we hide our light we collude in the overall cultural devaluation of our teaching vocation” (Bell Hooks).


The session included an interactive exercise that involved participation from the audience. As we went in we were given a pen and a piece of A4 paper with the words Why?, How?, A moment, and A model/metaphor on them as headings at equal intervals down the page. At the appropriate moment, Lisa asked the audience to look at the first question, gave some background behind it (why do you teach and what do you hope your learners will be able to do), and then each person wrote down their response to the question. The top of the paper was then turned over, and the audience had to swap papers with someone close by. This process was completed for the rest of the sections, swapping papers with people around the room after each one was completed. It was a really effective exercise as not only were the audience engaged and active, but it personalised the topic and the issues for them. To complete the exercise, 50% of the audience (in groups of 3) considered question 1 and selected 1 response that they felt resonated with them; 50% of the audience (in groups of 3) did the same for question 4. The microphone was finally circulated, with a nominated speaker reading out each group’s selection.

Examples of answers (all anonymous) on the sheet that I came away with were 1) Why? “I want my student to have the ability to survive the future”; 2) How? “One thing I do well is detailed instructions”; 3) A moment (of success) “Working with a group of Maori nursing students and having them tell me about the concept of Waiora, mindmapping what they shared on the board and offering it back to them as a structure for their presentation. Tutor as learner/learner as tutor.”; 4) A model/metaphor (for learning and teaching) “Reciprocal learners about life”.

I felt it was a shared reflective activity that appeared to work well. Each person had a ‘voice’ and had the opportunity to contribute to the discussion (although some people chose to opt out). Judging from the responses, people invested time and effort to respond meaningfully. The activity is one I will certainly use in Professional Development sessions with academic faculty, as well as at conferences when I present in the future!

Lisa summed up the exercise by revisiting the questions Why?, How?, A moment, and A model/metaphor. The second and third questions were designed to help celebrate who we are – to help us revalue our profession. At the heart of teaching is the relationship between a student, a teacher and a subject and this is where the models and metaphors are so useful. She suggests that metaphors do not describe reality, but that they create it. The image of the learner as consumer places the learner as passive, as a receiver…also, as a customer, the learner knows what they want, and that they are always right. Also, the metaphor suggests that the institution is the ‘shop’, and the person who sells the product rarely evaluates how the customer uses their purchase.

Other models that Lisa has collected from her other meetings, included co-learner, facilitator, mediators, mentors, guides, colleagues with a serious duty to care, and a co-traveller who sees things that otherwise the learner may miss. She argued that the whole dynamic between teachers and learners would change if the metaphor of consumer were abandoned and replaced with an alternative. The challenge is how this can be opened up for discussion in institutions, and unpicked with students, so that the customer model is left behind.

One analogy that a group of us discussed after Lisa’s session was that of a greenhouse (extending something that someone had written in the session about a teacher as gardener). We felt that the greenhouse idea included the notion of planting seeds, growth happening within each individual seedling, and the learning environment being safe (protecting from frosts), nurturing (the right balance of warmth and moisture, that is constantly monitored). It also allowed for differences (tomatoes not being the same as melons), and the notion that the seedling will eventually be transplanted to the outside world. Could be maybe stretching an analogy to breaking point and beyond, but it does focus on the fact that the teacher cannot do the growing for the learner, and that the learner has to be actively involved in the process.

Reblog this post [with Zemanta]

Sunday, December 13, 2009

ePortfolios: Recognition of Prior Learning and Communities of Practice


At the recent ePortfolio Symposium hosted in Melbourne and organised by Allison Miller of the Flexible Learning Framework, there was a great range of practical tips, and thought provoking presentations.

The blurb from the site advises: "More than 140 teachers, trainers, managers and ICT professionals attended the inaugural VET E-portfolios Showcase 09 (VES09) in Melbourne on Friday 16 October, either at the actual event or by participating in free online sessions, to learn from some of the most highly regarded e-portfolio researchers and leaders. Click here to view the full program."

I went to several presentations, and the notes were taken at two that I attended.


Annelieske Noteboom & Christine Cooper, (Challenger TAFE) - The ePortfolio Landscape.

A presentation about Recognition of Prior Learning (RPL), it was interesting that the presenters commented that more creative, flexible approaches to ePortfolios would be like having a cardboard box full of paperwork – as an RPL assessor you would never be able to sort through all the paperwork. She said that it was “all very nice to allow students to be creative, but it is not good to have students with an electronic ‘wheelbarrow’ full of stuff”, it needs to be organised". I certainly agree that every ePortfolio should have an element of organisation, and learners should be able to create different views from the collected and developed artefacts, planning, reflections etc. - one of which could be in line with RPL requirements, and easy for an RPL assessor to assess. The creativity is encouraged in the underlying learning and development, rather than, necessarily, in final ‘views’.

The presenters made another point around file consistency – but if the files are hosted online (and shared with only the assessor if privacy is an issue), then the files will stream, and this reduces problems with being able to view/listen to files.

Some initial reticence of participants to actually wear the camera glasses and talk their way through the processes they are undertaking. The camera glasses empower the learners though, as there is no-one there videoing them, but rather they take the video themselves, and if they are not happy with the result they can make another version…until they are happy with the result. Also have the ability to edit the video if they wish to…although I wonder if there are issues with verifiability if students are able to edit RPL videos? Does it matter? Would it not instead make it easier for learners to make sure, according to detailed rubrics that they are meeting requirements?


Gillian Hallan (Queensland University of Technology) - ePortfolio communities of practice: Local, national, global

This presentation (which can be listened to in full by clicking HERE) included highlights of the work that is required around sustainable practice and ePortfolios. Gillian also wanted to examine what was making existing CoPs work, and what stakeholders required to continue to make it successful.

The project started in December 2008 started with a literature review, and two AeP2 symposia and showcases. As part of this, ALTC was set up to as a group site, which includes an ePortfolio section. From March to July 2009 several research activities were undertaken along with a report around the research. Gillian made reference to Wenger’s definition of CoPs, before moving on to describing the CoP lifecycle, which goes through stages of inquire, design, prototype, launch grow and sustain, but many CoPs die just as they are emerging. The research Gillian was undertaking was around why some CoPs are successful, whereas others are not.

Some initial research findings suggested tat there was a 74% interest in a regional CoP, whereas there was a stronger (86%) interest in a national CoP, and reasonable interest in an international CoP (76%). Gillian mentioned that there are a number of communities formed around various aspects of ePortfolios, for example, there are some that are formed around supported learners through PDP (Personal Development Plans), and others that are formed around specific ePortfolio platforms.

The contexts for ePortfolio CoPs were: 1) pedagogy – 75%; 2) discipline based – 60%; 3) technology – 50%. Gillian went on to discuss the scope of collaboration for CoPs, in particular opportunities to collaborate, participate in special interest groups, disseminate information, and the develop further resources. Questions were posed around around whether an ePortfolio CoP should be organic or whether it should be managed. Almost 3/4 of respondents said that they felt there should be a funded CoP manager.

Some of the key factors for success that Gillian’s research identified were: “…our experience is that it needs a lot of mediation”; “Need a leader – need a community manager, without the work I do there is not a community “…a facilitator is critical – particularly around raising awareness”. The audience were surprised by the fact that a ‘manager’ was critical and cited social networking CoPs as an example of CoPs that are often not facilitated. Gillian then mentioned that respondents from the VET sector were much more in that social networking space, whereas many of the HE respondents were not in that same space – it did not form part of their day-to-day life. The large majority of the respondents in this research were from HE as this is the sector the research project was funded to support.

Success factors also included the fact that there needs to be a range of activities, and some face-to-face meetings can be refreshing. On the other hand, too much activity can be overwhelming and exhausting, and this is perhaps were a good facilitator comes in. Some of the main challenges identified included using the technology facilitator workload and community engagement. “Keeping engagement has been the largest challenge – feedback at events s generally positive but how” to measure the amount of activity?”. In a nutshell, there needs to be a balance. To achieve sustainability there are a range of internal and external drivers, including encouraging a broader interest in ePortfolios (e.g. employers), professional accreditation, cross-sectoral engagement, building, and extending national and international relationships.

Gillian concluded by saying that the final report will be a lot more detailed and will include a number of case studies that will present a richer picture of ePortfolio communities. In the meantime, it is important to continue to stimulate interactions and attract new participants, as well as a need to maximise the opportunities to build CoPs at the local, national and international levels.

ePortfolio platforms: Pros and cons


I was recently at an ePortfolio symposium in Australia, organised by Allison Miller. The first evening a number of ePortfolio platforms were showcased, and these are the notes, reflections and thoughts I recorded during the session.

Concord - http://www.concord-usa.com/scioware.htm

  • Use a lot of the 2nd/3rd year students as mentors to the freshers
  • Use ePF throughout their time at the college
  • One of the features of Concord, can package up different content

Assessment - some thoughts and questions

Not sure about the idea of locking down an artefact. How does that fit in with the notion of organic development? Is this a problem with the focus on assessment as an end product? Would learners still keep ePortfolios if they were not part of the assessment process? Perhaps one way to harness some of the key benefits of ePortfolios without tying them into summative assessment would be to have a ‘completion’ and/or participation grade. One of the things we found at Dubai Men’s College and at Unitec NZ is that it is not until learners have completed the first round or two of ePortfolio tasks, and have their heads around reflection and feedback, that they become aware of the learning benefits they are experiencing by completing their ePortfolios.

I would argue that once an artefact is locked down
  1. the learner loses their sense of ownership;
  2. they are dis-empowered; and
  3. possible ongoing learning experiences may be lost. Some students, in our experience, do continue to polish their designs and extend their ideas.

Desire2Learn ePortfolio - http://www.desire2learn.com/ePortfolio/
  • Focus on personalised learning
  • Very little common understanding of what an ePortfolio is and can be – is that necessarily a bad thing? However, important for programmes who are going to use ePortfolios to agree on what they are looking for, why, and how it fits in with the general learning outcomes of the programme
  • ‘Virtual collections’ – e.g. student studying vet science – take a photo; want to use it in 5 or 6 different contexts
  • Web-based technology (can be installed on own servers, or hosted by the provider)
  • Key areas: artefacts, collections, reflections, presentation
  • Pages built on HTML – edited through a WYSIWYG environment
  • Storage space – use the Web based storage facilities, and then link to it in the ePortfolio. Also enables external Web sites to be used as objects to be reflected on.
  • Competencies tool = learning outcomes (can take snap shots of achievement)…I wonder what this means. Completion? Assessment / assignment grades? Quality of reflection? Progress in an ongoing project?

Mahara - http://mahara.org/about/eportfolios
  • Multiple, tailorable views, using the same artefacts, for specific audiences
  • To set up templates for assessment use a control group (closed group, open group, control group). Can put start/stop date as part of the design when the view can be accessed
  • With Mahara you can release an artefact back to the student to continue working on

Assessment - some thoughts and questions

Is there the danger that artefacts for ePortfolios are only being created for assessment – glorified essays with illustrations? Do some of the tools focus on tying-down students? I wonder if there isn’t the opportunity instead, to personalise assignments (for authenticity as well as effective learning) in a way that also mimics authentic contexts, or immerses learners in situations that encourage the application of analytical, problem solving skills to authentic problems or issues.

Pebble Pad - http://www.pebblepad.co.uk/
  • Assert that ePortfolios can be restrictive, and therefore, that ePortfolios are only one part of Pebble Pad
  • Pebble Pad is therefore described as personal learning space

Tuesday, November 10, 2009

Where do I Start? Integration of ICTELT into blended curricula

Abstract:"The potential of information communication technology (ICT) to enhance learning is under-utilised even though there are numerous process models and frameworks have been developed to assist in the design and / or adaptation of curricula. Issues with many existing design models are complex and range from the pragmatic, such as resource requirements, skills and ownership of a project, to models being mechanical, inflexible, hierarchical and / or impenetrable.

This session starts by exploring the learning and teaching value ICT can add to curricula, and then describes a process model and framework that I have adapted from existing examples. The accessible, scaffolded approach described is appropriate for very small teams or individuals working with few resources to develop resources ideal for instructors interested in blended learning and/or distance approaches.

The pedagogical underpinnings of a design process are outlined, in which practitioners identify a teaching and learning problem and assess whether ICT could enhance learners experience of new or existing programmes, modules, units, sessions, or learning objects. Guiding questions are posed to help support the process, and an iterative practice is encouraged whereby a design is developed, piloted, evaluated, revisited, modified and re-evaluated over time, with recognition that the practitioners experience, skills and attitudes are likely to shift.

The practical application of the model and framework is illustrated through an example developed for use in Moodle at UnitecNZ.

The model and framework have yet to be piloted and MoodleMoot will offer a hands-on opportunity for participants to experiment and evaluate the tools in this session by identifying a resource or session that they would like to adapt. Then, working collaboratively, in Moodle they will work through the first part of the process model by completing the framework. Feedback and comments will be gladly accepted."

Associated sites: Slideshare: http://www.slideshare.net/hazelowendmc/where-do-i-start-integration-of-ictelt-into-blended-curricula-presentation

Wiki: http://efest2008collaboratingbydesign.pbwiki.com/

Please cite as: Owen, H. (2008, October 8-10). Where do I Start? Integration of ICTELT into blended curricula. Paper presented at the MoodleMoot NZ 08, Eastern Institute of Technology, Hawkes Bay, New Zealand.

Information literacy skills - disipline-specific, online tutorials




You might like to check out these online, discipline-specific tutorials from the 'Virtual Training Suite'. The blurb on the site describes them as "a set of free Internet tutorials to help you develop Internet research skills for your university course", and goes on to say that "All our tutorials written and reviewed by a national team of lecturers and librarians from universities across the UK". The estimated time for each tutorial is one hour of self-directed study.

Friday, November 6, 2009

Laptops in the Classroom - fundamental questions


The other day, two colleagues asked if I knew of any videos of sound practice around students using laptops in the classroom. Actually, there appears to be a dearth of such material. I have some word pictures I could paint around the experience I have had in Dubai and at Unitec, and I am sure Thom could do the same...but not quite as visual :-)

I did however, after some hunting around come up with the range of videos:
  1. http://www.youtube.com/watch?v=uzJDuBlIrHc
  2. http://www.youtube.com/watch?v=mOYg5ZMo-mA
  3. http://www.youtube.com/watch?v=Pj45RmhaCQM
  4. http://www.youtube.com/watch?v=TkXeNR52w_o
  5. http://www.youtube.com/watch?v=1m8Ax5NUHhg

While watching the video, for me a number of fundamental questions arose:

  • What are the assumptions around learning indicated by what is said by students and teachers/lecturers in these videos?
  • What are the assumptions around teaching indicated by what is said by students and teachers/lecturers in these videos?
  • Was there much evidence of 'real' collaboration and co-construction of knowledge?
  • What did the educators do to facilitate 'real' collaboration and co-construction of knowledge?
I could go on, but I think four questions are about right! Thoughts, ideas, issues, concerns...?

Monday, November 2, 2009

ICTELT in Action: Applying ICT Enhanced Learning Programme Design

This video is an extract from a workshop that was facilitated by Diana Ayling and Hazel Owen with the Business Department at Unitec NZ. The workshop was part of an initiative initiated by the Business Department to revisit the programmes that they offer students, and the learning experience. Formal and informal sessions focussed on aspects such as:
  • Course design and lesson planning
  • Facilitation of learning
  • Assessment
  • Evaluating teaching and learning

A sense of the dynamic discussions that have been ongoing is captured, as well as a feelings of excitement, seeing potential, anxiety, and concerns around pragmatic issues.

The extract features discussion around the results of the ICTELT survey that participants had completed before the session, as well as dialogue around blended learning in general, and what a course that uses the ICTELT design mindmap as a foundation may feature.

(If you would like to find out more about the work that underpins some of the workshop and the ICTELT mindmap, please feel free to visit the ICT Enhanced Learning and Teaching wiki.

A discussion: ePortfolios - how are they supporting 21st Century learners?

"A discussion: ePortfolios - how are they supporting 21st Century learners?" is the recording of a conversation that occurred at the eFest meets Teaching and Learning Conference (http://efest-teach-learn.ning.com/). The discussion covered a range of themes, issues and potentials, with particular reference to the tertiary education and vocational education and training sector.

The discussion was facilitated by Justin Sampson from Ako Aotearoa, who plays a key part in the ePortfolio Community of Practice hosted by Ako (http://akoaotearoa.ac.nz/communities/eportfolios-new-zealand).

The combined conference eFest meets Teaching and Learning Conference was run from Wednesday 30 September to Friday 2 October 2009 and was hosted by UCOL in Palmerston North, NZ.

Friday, October 30, 2009

Pedagogy, Policing or Preventing Plagiarism? Experiences with facilitating Professional Development and Turnitin


This was a paper Vickel Narayan and I presented at the eFest meets Teaching & Learning conference, UCOL 2009. It was an interesting experience at a vibrant conference. A time for firsts, it was Vickel's first time to present and write an academic paper for a conference, and it was the first time that we had written and presented together. I felt that we both learned a lot on the way, from each other, from the feedback we gathered from the audience (see below), and from reflecting on how well it went on the day.

It was an interesting audience, and it was great to have much more of a discussion underway throughout the session. There appeared to be a lot of sharing of experiences, strategies, and problems faced, which saw the audience fully engaged. Throughout the session there was also a couple of people Tweeting about main points, and I found this was a great way, afterwards to see how people had interpreted what had been said, and what they saw as central points. It was also great to find out, for example, that the mindmap I had made building on previous work I'd done in 2006, was in fact useful and did illustrate some key factors behind why learners plagiarise.

All in all, judging from the comments, the discussion, and the feedback, I think the audience went away with some new ideas and approaches, as well as affirmation that they 'not alone'! :-)

To access the accompanying handout: http://www.scribd.com/full/20133056?access_key=key-2kh798rentwwpywxfd9l

To access the full paper: http://www.scribd.com/full/18077894?access_key=key-2d8r0452hsbwlgdffwx7

Feedback

  • Excellent; general overview; positive approach to manage plagiarism
  • Session was great. Just enough time to have our discussion!
  • Liked the suggestion: getting students to have constructive dialogue about plagiarism
  • Love the handout w the links. Thanks heaps. T
  • I enjoyed the session but needs longer time for discussion
  • Plagiarism is a lot more than I realised
  • Important - see a way to move from punitive to formative
  • "Use a free tool for formative use" - this was a new idea for me! Thanks for all the links - I will use
  • Good to see Turnitin as a formative tool rather than punitive
  • Turnitin as a tool - not the answer
  • Thanks for the references
  • Cultural issues seem to be the main factor but I value the deterrence factor and want to improve arguments

Abstract: Plagiarism is a global issue that needs to be addressed by all educators and learners. This paper considers a simple definition of plagiarism, and then briefly considers reasons why students plagiarise. At Unitec NZ, Te Puna Ako: The Centre for Teaching and Learning Innovation (TPA:CTLI) is working closely with faculty, managers, student support services and library personnel to introduce strategies and tools that can be integrated into programmes and curricula whilst remaining flexible enough to be tailored for specific learners. The authors therefore provide an overview of one of the tools available to check student work for plagiarism - Turnitin - and describe the academic Professional Development (PD) approaches that have been put in place to share existing expertise, as well as help staff at Unitec NZ to use the tool in pedagogically informed ways, which also assist students in its use. Evaluation and results are considered, before concluding with some recommendations. It goes on to theorise how blended programmes that fully integrate academic literacy skills and conventions might be used to positively scaffold students in the avoidance of plagiarism. Conference participants will be asked to comment on and discuss their institutions' approach to supporting the avoidance of plagiarism (including the utilisation of PDS and other deterrents), describe their own personal experiences, and relate the strategies they employ in their teaching practice and assessment design to help their learners avoid plagiarism. It is planned to record the session so that the audience's narratives can be shared with other practitioners.

Please cite as: Owen, H., & Narayan, V. (2009, 29 Sept - 01 Oct). Pedagogy, Policing or Preventing Plagiarism? Experiences with facilitating Professional Development and Turnitin. Paper presented at the Teaching excellence - excellence in teaching: Teaching and Learning Conference Meets eFest 2009, Universal College of Learning (UCOL), Palmerston North.



Thursday, October 22, 2009

Web 2.0 ePortfolios that work for both students and educators: Strategies and recommendations

To access the accompanying handout: http://www.scribd.com/full/20963840?access_key=key-tjhoooneoyc6p12igkx

Abstract:
The VET ePortfolio Roadmap was released in June 2009 to provide guidelines, specifications, and strategies for implementing ePortfolio initiatives. The Roadmap was published, in part, as a response to the increasing interest in the potential of ePortfolios to improve the Recognition of Prior Learning process, and expedite work-based learning, apprenticeships, and traineeships. Previous research studies into learners' use of ePortfolios endorse this response, suggesting that their levels of engagement, creativity, and feelings of empowerment are enhanced, thereby increasing retention and success. It all sounds extremely promising...but what does it actually 'look' like for students and educators? How are learners, practitioners and other stakeholders actually engaging with ePortfolios?

In this paper I have three main aims. The first is to provide some background by referring to an early initiative that was implemented between 2003 and 2006 with Foundation students at Dubai Men's College (DMC) where the students created a Career ePortfolio as part of an integrated Computer, Research Skills and Projects Course. The ePortfolios, however, were not interactive, were rather 'static', and the final artifact was primarily produced for assessment rather than self-reflection and development. Since this and similar early initiatives, the introduction of Web 2.0 social software elements to ePortfolios has helped realise additional benefits, including improved reflective practice, augmentation of the quality of final artifacts, and a heightened awareness of purpose and audience. As such, the second aim is to explore recent work with Web 2.0 ePortfolios with students and faculty at Unitec NZ (a multi-sector education institution in NZ), and some of the associated findings and implications. Finally, I will draw the threads together to discuss a number of key strategies and recommendations for the effective implementation of Web 2.0 ePortfolio initiatives, including targeted Professional Development for staff, and scaffolding and guidance to assist the students with self-reflection, collection and selection of evidence of achievements, while also fostering their personalised and creative life-long learning journeys.

Please cite as: Owen, H. (2009, October 16). Web 2.0 ePortfolios that work for both students and educators: Strategies and recommendations. Paper presented at the VET E-portfolios Showcase 09 - learning for life.

Monday, October 19, 2009

Web 2.0 ePortfolios that work for both students and educators: Strategies and recommendations

*NB: currently there is about 2 minutes of dead space at the beginning of the presentation. I hope to edit this out later.

To access the accompanying handout: http://www.scribd.com/full/20963840?access_key=key-tjhoooneoyc6p12igkx

Abstract:
The VET ePortfolio Roadmap was released in June 2009 to provide guidelines, specifications, and strategies for implementing ePortfolio initiatives. The Roadmap was published, in part, as a response to the increasing interest in the potential of ePortfolios to improve the Recognition of Prior Learning process, and expedite work-based learning, apprenticeships, and traineeships. Previous research studies into learners' use of ePortfolios endorse this response, suggesting that their levels of engagement, creativity, and feelings of empowerment are enhanced, thereby increasing retention and success. It all sounds extremely promising...but what does it actually 'look' like for students and educators? How are learners, practitioners and other stakeholders actually engaging with ePortfolios?

In this paper I have three main aims. The first is to provide some background by referring to an early initiative that was implemented between 2003 and 2006 with Foundation students at Dubai Men's College (DMC) where the students created a Career ePortfolio as part of an integrated Computer, Research Skills and Projects Course. The ePortfolios, however, were not interactive, were rather 'static', and the final artifact was primarily produced for assessment rather than self-reflection and development. Since this and similar early initiatives, the introduction of Web 2.0 social software elements to ePortfolios has helped realise additional benefits, including improved reflective practice, augmentation of the quality of final artifacts, and a heightened awareness of purpose and audience. As such, the second aim is to explore recent work with Web 2.0 ePortfolios with students and faculty at Unitec NZ (a multi-sector education institution in NZ), and some of the associated findings and implications. Finally, I will draw the threads together to discuss a number of key strategies and recommendations for the effective implementation of Web 2.0 ePortfolio initiatives, including targeted Professional Development for staff, and scaffolding and guidance to assist the students with self-reflection, collection and selection of evidence of achievements, while also fostering their personalised and creative life-long learning journeys.

Please cite as: Owen, H. (2009, October 16). Web 2.0 ePortfolios that work for both students and educators: Strategies and recommendations. Paper presented at the VET E-portfolios Showcase 09 - learning for life.

Sunday, October 4, 2009

Motivation, agency and purpose

Image source

Our department is just about to have it's strategic planning day, so I've been doing some thinking and research around what works with individuals and teams, and came across this really thought-provoking TED talks video by Dan Pink: http://www.ted.com/talks/dan_pink_on_motivation.html.

One of the key notions Pink discusses is that extrinsic motivation only works with tasks with a low cognitive load, whereas creative, right brain tasks require intrinsic motivation. Motivation is also enhanced by a clear sense of purpose. Pink's argument is that business is being run on a model that has been proven by research that it doesn't work (reward), and that it needs to shift to a model that supports and enables creative motivation, trust, and teamwork. After watching his video, I went away convinced, and will now have to begin a quest to shape my own working environment as Pink recommends :-)

Pink also has a blog, where he has posted an entertaining Dilbert about extrinsic motivation, a discussion around the fact that money cannot buy motivation (but that a fair wage needs to be agreed), and a critique of the importance of sabbaticals and creativity.

The running notes from the TED blog are as follows:

"Dan Pink, once a speechwriter for Al Gore, is now a career analyst beginning a revolution in the workplaces of the world. This morning at TEDGlobal he begins by noting that a little over 20 years ago, he did something that he regrets. He went to law school. He didn’t do very well. Pink jokes that he graduated in the part of his class that made the top 90 percent possible. He never practiced law a day in his life as he wasn’t allowed to. But today, against his better judgment, he says, he wants to use some of those legal skills. He wants to make a case for rethinking how we run our businesses.

Pink shows a slide title "The candle problem," a psychological experiment created by Karl Duncker in 1935. A person is brought into a room and given a candle, a box of thumbtacks and matches and asked to attach the candle to the wall so that the wax doesn’t drip on to the table. The person who can solve the candle problem is one who, rather than seeing the box as receptacle for the tacks, sees it as something that can be used in the solution. The box is tacked to the wall and the candle placed on it.

This experiment is used to learn about incentives, Pink explains. Two groups of people are offered the problem -- the first group is simply timed and the second group is offered rewards. It takes the second group three and and a half minutes longer than the first group, on average, to solve the problem. "That’s not how its suposed to wrk! I’m an American. Incentives work!" Pink exclaims. But, he says, this experiment has shown that incentives actually dull thinking and block creativity and he notes that this is not an aberration. It’s been shown over and over again. It’s one of the most robust findings in social science and also one of the most ignored. There’s a mismatch between what science knows and what business does.

Another experiment was done with the problem presented in a slightly different way. Th tacks were taken out of the box, and then the incentivzed group did much better than the other. Pink says this is because it’s an easy problem. For these types of tasks of narrow focus, where you can see the goal right there, rewards work really well.

However, he points out that around the world, white collar workers are doing less of this second type of work and more of the first. Narrow tasks have become fairly easy to outsource and to automate and right-brain conceptual tasks have become more important. Everybody in this room, Pink says, is dealing with their own version of the candle problem. And for those people the if-then rewards don’t work. "This is not a feeling. I'm a lawyer, I have no feelings. This is not a philosophy. I'm an American, I don't believe in philosophy," he says. This is a fact, Pink asserts.

He draws on the a study by Dan Ariely and his colleagues. Ariely et al found that once the given task in one of these experiments was only a mechanical skill, rewards would mean better performance, but if any rudimentary cognitive skill was needed, a larger reward would mean a worse performance. The study was retested in India to control for cultural differences and they found got the same results. Studies at the London School of Economics have also found that financial incentives can result in a negative impact on performance.

So, Pink says, to get out of the messes of the 20th century, we don't need to do more of the wrong things. We need a new approach, one that includes three basic elements: Autonomy, mastery and purpose. These are the building blocks of an entirely new operating system. Today, he says he's going to talk about autonomy. The traditional notions of management are great if you only want compliance, he explains. But for creative thinking, we have to approach things differently.

He points to the software company Atlassian -- a few times a year, they tell their engineers to go off for 24 hours and work on anything that is not their regular job. Then they all come back together and present their work. They call these Fedex days, because they have to deliver something overnight. Atlasssian has taken also implemented the 20 percent time rule that Google has, where employees can take that 20 percent of their time at work to work on whatever they want. Pink says that about half of Google's products have come from that time.

Pink also advocates results only work environments (ROWE) where there are no schedules, people don't have to work in the office, employees can work wherever and whenever they want and meetings are optional. When companies implement ROWE policy, he says, productivity always goes up and turnover goes down.

For more evidence, he discusses two different models that were posed for creating a digital encyclopedia. The Microsoft model which included hiring researchers and experts and extensive planning, and the Wikipedia model where people would participate because they were interested. Pink asserts that 10 years ago, you could not have found two economists who would have said that the Wikipedia model would work better, but it does.

Science knows that motivators only work to solve narrow problems, Pink declares, but they destroy creativity. Maybe, he says, if we can increase productivity in solving the candle problems everywhere, we can change the world."

Thursday, October 1, 2009

The Web: End of the classroom era?



The “The Web: End of the classroom era?” debate today at the eFest meets Teaching and Learning Conference, was well-attended. Initially the rules around the debate were established: panel of 6 people, 3 affirmative (in agreement with the statement) and 3 negative (in disagreement with the statement). Both panels were seated on the stage. The audience and a red and green voting paper. There was an adjudicator and a timekeeper. Each panel had 3 minutes each to speak.

The first speaker was from the affirmative team, and began by giving some statistics around online learning. Learners in online courses spend more time on task than in purely face-to-face contexts, and achievement of learning outcomes is higher. The second speaker from the panel in disagreement, responded by coming from the ‘human’ aspect – human teachers for human learning. It was interesting that face-to-face was seen as a positive approach for emotive factors, and being able to ‘see’ where learners were at, and what they were feeling. There appeared to be no understanding around the range of tools that can facilitate these same factors, the fact that somehow face-to-face learning is better – that there is no boredom, more engagement, more opportunities to contribute. Joyce was quick to challenge these ideas, pointing out that in face online it can be more democratic, that more learners have a voice, can create an identity. She recommends the You Tube video: Mr Winkle awakes. The Internet releases the teacher from the information transfer model and frees them up to debate.

The next speaker did not rebut any of the previous points, saying that she felt nothing had been said that needed to be rebutted! She feels that the Internet caters for the trivial, and referred to Andy Warhole’s 10 minutes of fame. The Web lets inadequate, foolish people express their thoughts, and the Web can deceive. You can’t trust the Web and there are perils and traps for young players. The next affirmative speaker chose to rebut the previous speaker’s points. He pointed that Ivan Illich foresaw the Internet. Schools and classrooms are industrial strength learning and that this is not the sort of learning we need. He quotes Illich’s book around conviviality. Learners are reliant on teachers and the system that makes education work. Learning is what happens on the Internet, and that will usher in the post-industrial age.

The final person from the disagreement panel, Colin Cox made a comment about the fact that he had not heard the previous speaker as he was Tweeting. He than gave an overview of his flight from Auckland to Palmerston North, pointing out that pilots who had learned in a simulated environment would not have the skills to really fly a plane.

Next stage was to throw things open things open to the audience for questions and comments. The first person from the audience made the point that simulators have been used to train medical professionals and pilots for years; that dentists were being trained in second life. The rebuttal made the point that the classroom is as big as the world – which played directly into Joyce’s hands – who pointed to the title of the debate!!

A question from the audience asked why eFest took place in a classroom on Tuesday. It was argued that eFest in fact had started months before in wikis, through Twitter, in Skype meetings – in a collaborative, globally diverse participants. The one laptop per child initiative enables children to connect to each other, even if they are not able to connect to the Internet.

One of the audience pointed out that she felt nervous about contributing to the discussion, but if she were online she would be able to contribute more comfortably. She poses the question – are teachers motivated by popularity, and their ‘ego’, when students appear to ‘love’ them?

A final comment from the audience pointed out that he could’t imagine his kids picking up the skills they currently have in an online environment…like kicking a ball.

A final vote from the audience was requested after a humorous summary of the main points made in the debate. It was a close call. The adjudicator declared the debate a draw given the quality of the debate and the spirit behind it!!!

I heard the people behind me say that it had been the best part of the conference. It certainly provoked passionate responses that showed the depth of commitment and feeling educators invest in their ‘calling’.

Talent devlopment




“If people knew how hard I had to work to gain my mastery, it would not seem so wonderful at all” Michelangelo

This post describes a session given by Colin Cox at the eFest meets Teaching and Learning 2009 on October 1st 2009. It focussed on talent, motivation and myelin! It posed the questions: do you do what you do because you have natural talent? Mystery about how to people have talent, what is it, and where does it come from?

A session that explored what the components that make talent and how you ‘grow’ talent. It was a quite interactive session, which involved for example speaking to the person sitting next to you and asking what they felt talent is, and then collecting some of the responses from the audience verbally. Only about 30 seconds per question was given, but the replies indicated that it was just long enough for people to formulate ideas and replies. The answers indicated quite a cross-section of opinions around the subject of talent, its source, and how/if it can be enhanced. Some key ways that were seen as effective ways of enhancing the potential of learners in education, including coaching, self-belief, practice, mentoring, commitment, opportunity.

Some examples of people with talent were Lance Armstrong, Tiger Woods, Jerry Rice, Shane Cameron, and this then moved to a discussion of ‘nature or nurture’. If it isn’t nature then the conclusion was that it would exclude a lot of people. The question was also raised about excellence or mastery in any endeavour and if it was about innate talent, IQ, memory, physical prowess, motivation, practice or a combination.

The example of Tiger Woods was unpicked, and the notion of the fact it was hard work and role models that made the difference as opposed to genetics (although genetic potential does determine height, weight, and muscle mass). By his first major competition at the age of 17 he had already been practising golf since he was 1 ½ years of age.

The next area that was explored was short-term memory (cited the 1978 Carnegie Mellon University study). Cox role=played the part of SF who was able to remember 22 digits after hearing them only seconds before – 1 per second. The amount of effort was physical and emotional. With 250 hours of practice SF was able to recite back 88 digits having only heard them once – 1 per second. The short-term memory is something that can be expanded and extended with practice. The records are now being broken regularly, from 102, to 3 card decks (52 cards in a deck). It was emphasised that SF had average ability.

IQ was the next subject to be put under the spotlight. A study around sales people and horse racing was cited, and looked at who was the most successful, given the information that they have, at predicting results. IQ was not a predictor of success – the lawyer (IQ 118) was not as successful as a construction worker (IQ 85), with the construction worker using more complex formulae, and a success rate double that of the lawyer.

Cox gave an overview of the ‘Myelin secret’ – it’s not about neurons and synapses it’s about Myelin, the neuron insulator. The more you do an activity, the more it insulates the neuron, which takes a neuron from basic functionality to ‘super’ functionality that enables quick responses, thinking and interconnections. Neurons are stimulated by outside actions and influences; the first time you do something neurons fire slowly and are relatively ‘uncoordinated’ – the more something is done, the more the myelin insulates the neurons, and the better the performance of the neuron. Einstein’s brain was the same size as the average person but it had a lot more white matter – myelin.

As an overview, Cox concluded with a discussion of:
1) passion; passion maximises talent because talent requires effort and time, and is not always fun; Where talent meets talent, the person becomes unstoppable. Initially, the passion is not there; there is the requirement for a role model who encourages people to try something. Once a person tries something and realises that they are quite good at something, then this can translate to passion that helps you overcome difficulties and hardships. It helps when the role model is also passionate.
2) practice; it is not any old practice that makes the difference, it is ‘perfect practice’. The elite don’t just practice, they do deliberate practice that is designed to improve performance, and helps the individual identify specific elements for improvement. What meaning has this for students? The comfort zone is not your friend! Life needs to include challenge, things that push you outside of your comfort zone – things you do not know how to do. However, the ‘panic zone’ is also a negative space where a person is pushed too far, which often results in an individual giving up and/or withdrawing. Each student has a different range of comfort, learning and panic zones, so learning experiences need to be tailored to the individual. Practice is the ‘mother’ of skills and includes observation, imitation and repetition. It includes focussed work that builds myelin. Anders Ericsson says the 10-year (10,000 hours) rule appears to be consistently accurate – i.e. it takes 10 years to master something, whether it is playing a musical instrument, playing a sport, excelling in a career.
3) Feedback is vital when you are practising.

A thought-provoking session, with a lot of application for education, and helping learners meet their potential.

Wednesday, September 30, 2009

Association for Authentic, Experiential and Evidence-based Learning

Not only are the conversations on the Association for Authentic, Experiential and Evidence-based Learning (AAEEBL) site lively, but there are some great resources being shared that members have developed. For example, this list of resources around reflection for learning compiled by Helen Barrett. The particular focus is ePortfolios, and it is a good place to start if you are looking for ideas and would like to join in the conversation.

Wednesday, September 16, 2009

Evaluating feedback and collaborating at conferences

Bettina Schwenger and I recently presented at The student experience: HERDSA 2009 in Darwin. As part of the presentation we asked participants to add details to the mindmap pictured below - the original mindmap only included the nodes that have images. Participants collaborated in pairs or small groups to add to the mindmaps, and we collected all of their contributions at the end of the session. The results have been collated and added to the original mindmap, which has created an insightful visual resource into beliefs around Professional Development and the embedding of Literacy, Language and Numeracy into programmes.

At the end of the presentation we asked for evaluation of our session via sticky notes that were left on the door as people left, and received some very positive comments which have been collated below. In particular, it was reassuring to see the comments about the transferability and generalisability of the model in particular, and the approach to PD as a whole.

Resources

Feedback from the participants at the session:
  • Well presented and food for thought in dealing with resistant teaching staff. Thank you
  • Great project. Well done
  • Great model (transferable to any T& L context or group) for meaningful PD engagement of academics. (Could be replicated with students)
  • Generalisable model. Thanks
  • Inspiring. Good stuff
  • Provoking. Thanks for some great ideas
  • Thought provoking.
  • Breaking down barriers
  • Interesting pictures on slides
  • Nice symbolism
  • Used accessible terminology
  • Excellent. Good job. Well done.
Mindmap
Link to the 'live' mindmap (pictured below): http://www.mindmeister.com/23798819/building-capability (you will need to access the online version to be able to view the full mindmap :-) )

Intrinsic motivation: Seven Great Walks in seven days....


Not sure if this is the right forum for this, but thought it was not only an awesome example of planning an incredible feat, but also something that highlights what is potentially possible when we are motivated and engaged.

The introduction to the site explains that "The 7 in 7 is something that has never been attempted before. This will come as no surprise to most people – it is after all a pretty crazy thing to do! But for its creator, and main protagonist, Mal Law it is the opportunity of a lifetime to “make pain my friend”, experience New Zealand wilderness at its best, and most important of all, raise at least $50,000 for a great cause – The Leukaemia & Blood Foundation of NZ. The Challenge is a world-first attempt to RUN New Zealand’s 7 mainland ‘Great Walks’ in just 7 (consecutive) days. Completing it will be the equivalent of running 9 off-road marathons and climbing Mt. Cook twice, all within just 7 days."


I found out about the planned epic when a couple of dear friends, whose son Matthew was diagnosed with leukaemia in his first few months of life, mentioned the run. Matthew not only survived after much treatment, care and love, but is now a strapping 2-year-old. Jeff Greenwood (Matthew's dad) is joining Mal for the Heaphy and the Kepler; you may spot him training around Auckland...complete with baby buggy! :-)

Motivation is a strange thing, comes from many sources, and can help people achieve what appears to be impossible. Long live motivation!!

Tuesday, September 15, 2009

Building communities in Te Hononga: Unitec Department of Architecture Maori Studio


A sunny day and a pleasant stroll across campus set the scene for the Ning session Ed and I facilitated today with students and academic faculty from Te Hononga: Unitec Department of Architecture Maori Studio. Already keen Moodle users, students and faculty were looking for ways that students could be empowered to create their own space, add their own multimedia, try out ideas and designs in a safe space, receive feedback, work in groups on projects, and keep online journals. In response, Rau Hoskins and I set up a Ning site and today's session was to explore and discuss some of the potential uses of the site, and answer questions.



Rather than have a 'how to...' session students have been encouraged to access Anthony Armstrong's great online resource...a step-by-step Ning tutorial (with accompanying .pdf should you wish to download key points).


A decision was made to keep the Moodle site as the showcase for work and projects everyone is collaborating, and to use the Ning site a test and prototype area, as well as somewhere to get to know each other better. The Ning site is already buzzing with student activity, which is great. It will be fascinating to see the momentum build.

Friday, September 11, 2009

Knowledge creation/management and communities....


This is an informative, insightful interview conducted by CIO Insight with Etienne Wenger entitled Expert Voice: Etienne Wenger on Knowledge Management. I felt it, in part, addressed a couple of questions that arose in the Ning workshop Vickel and I facilitated with the Department of Management and Marketing this week on why one might cultivate communities between colleagues, learners, and a combination where appropriate.

The interview covers questions such as 'Who has used technology well to support a community?', 'Is there a risk of overmanaging communities?', and 'So the value of communities extends beyond the creation of knowledge?'.

The introduction to the interview reads:

"How is knowledge created? And what's the best way to put that knowledge to use? Those two questions have long been central to the work of Etienne Wenger, an independent researcher, consultant and author. The 49-year-old Wenger, a native of Switzerland, has spent his career spreading the concept of "communities of practice"—groups of people within organizations working together to create and apply knowledge."

Thursday, September 10, 2009

Enhancing online discussions - some ideas from the blogosphere...



Good afternoon. I thought you may be interested in this blog post about encouraging learners to 'talk' in online discussion forums. It is entitled "Say what? Promoting discussion in online courses", and has some interesting tips and wrinkles.

The introduction from the blog post is as follows: "How can we get learners to talk in online discussions, and how can we get the chatty students to shut up? The results have been pretty disappointing so far. Most instructors set up a discussion forum, and ask students to post an original post and to comment on two other posts. But instructors complain that students are doing the minimum and the discussions aren’t exciting. But what are they assessing? They’re just counting how many posts each student gives. The discussions, hence, are seen as tedious busywork."

Management, Marketing and Ning-ing


Today, on the auspicious day of 9th September 2009 (09.09.09) Vickel Narayan and I facilitated a workshop with the Department of Management and Marketing as a requested follow up session to a more generic Web 2.0 session, which Vickel had previously run with the group. Prior to the workshop we worked with the department to identify outcomes that they would require from two Ning sessions (number 2 in October) - this helped ensure that participants voices were 'heard' beforehand, and also that we didn't run the session that we thought they wanted, as opposed to the session that they actually wanted :-)

Session 1, today's, was a hands-on session that focussed on utilising a Ning space for their Department which could be used as a way to build more of a community, communicate and share. There was also conversation around negotiated guidelines, which they would later need to discuss. The agreed guidelines would be used to shape interactions they all felt comfortable with within the online forums, the 'identity' they shared with colleagues, and the content that was considered appropriate.

By the end of the session all participants had joined the Ning, added a profile and photo, posted to a forum, written and posted a blog reflection, and searched for, embedded and shared a video.

Session 2 will focus on using Ning with learners.

After a start where there was some trepidation, the room was filled with laughter and talking. One participant commented in their blog at the end that they were really happy that everyone was involved and having a bit of fun! A further participant came up and said she had been really anxious before attending the session as she felt she would not be able to do or accomplish anything. In fact, she had achieved a lot and was really over the moon.

Watch this space for the next installment.

Friday, September 4, 2009

Looking for images you can use without violating copyright? What is Creative Commons?



At a meeting around designing online courses the other day, the question was posed around which resources could be sourced from other sites without violating copyright. I gave a very brief overview of Creative Commons, but there are some fantastic sites out there that explain it all, O so much better than I can :-). Creative Commons is one such site where you will find a wealth of information and resources (including a place to license your own online materials). For those rushed off their feet I located a video from their site which clearly explains Creative Commons: http://ictelt.blogspot.com/2009/06/what-is-creative-commons.html.

Also, should you be looking for copyright safe images Techradar.com have located and reviewed 12 sites where you can search for free images: http://www.techradar.com/news/internet/web/12-essential-sites-for-great-free-photos-624818

Tuesday, September 1, 2009

Design and scenario: Web 2.0 ePortfolios for students

Image source: http://www.flickr.com/photos/waltstoneburner/3375027629/

I have been working on developing a scenario and design mindmap that explored some of the potential of Web 2,0 ePortfolios for learners. Given that I am currently working with Unitec NZ I decided to base my fictional learner, Chan Sook, at this institution studying The Bachelor of Business (Accountancy).

You can access the mindmap at http://www.mindmeister.com/25663129/chan-sook-learner-eportfolio and the accompanying scenario (which includes a list of Web 2.0 tools she uses, and how she uses them) at http://docs.google.com/View?id=dcqj5jv4_50cfd7tbfw.

Any comments, feedback or ideas that you may have would be gratefully received :-)


Participation in Sports....



Yesterday, Monday 31st August, Diana Ayling, Ed Flagg, Trisha Hanifin and I made our way over to the Sports' Department building and having located important things (like the kitchen to make tea) we set about re-arranging the furniture so that everyone would be able to sit around the table and share laptops between them. We promptly fired up the laptops, opened the browser and the Ning that had been set up specifically for Sports. Several people had already added their profiles and a sense of community was being built by the addition of photographs from.


Diana artfully facilitated the session as I typed notes like a mad woman and fielded the occasional request for technical assistance. The session started with introductions and each person chose a category off the board which they had to describe (around assessment, evaluation, and design etc). The activity encouraged a lot of engagement, with a healthy mix of competition, encouragement and humour.

After an overview of some of Fink's ideas, Diana encouraged everyone to login to Mindmeister and start putting together the key aspects of a Level 5 course. There were some of the 'so what do we do? ' questions, but then everyone got up to speed. Someone commented that they were chuffed that we were using the same software that was used to create some of the earlier graphics that Diana had used. There was a continual buzz and exchange of ideas, with everyone on task and engaged - in fact everyone was so involved that Diana had to work hard to get everyone back into focus.



Diana wrapped up by giving everyone a quick tour of some of the key resources in the Ning and encouraged people to go and have a look at some of these resources, thereby modelling some of the uses of the Ning. She in particular introduced reflective evaluation, introducing the blogs and targeted feedback, and described what the participants were going to do as 'homework' before the next session.

All in all there was a great mixture of fun, inquisitiveness, and professionalism, which Diana worked to encourage. Roll on the next session which is on 22nd September.

Thursday, August 27, 2009

Study Finds That Online Education Beats the Classroom....


An article from The NY Times has summarised the findings of a 93-page report on online education, conducted by SRI International for the Department of Education (US). The key findings are:

"Over the 12-year span, the report found 99 studies in which there were quantitative comparisons of online and classroom performance for the same courses. The analysis for the Department of Education found that, on average, students doing some or all of the course online would rank in the 59th percentile in tested performance, compared with the average classroom student scoring in the 50th percentile. That is a modest but statistically meaningful difference."


I must admit that I haven't as yet read the report in full. However, yesterday (26th August 2009) I participated in a really thought-provoking session run by The Centre for Teaching and Learning Innovation: TPA around literacies and the huge range of 'texts' we interact with and produce...digital, visual, audio, written, academic, multimodal. The conversation covered a variety of topics, and as the session progressed a couple of things occurred to me. Firstly, a lot of the participants seemed to think of digital literacies as basic ICT skills (i.e. turning on a computer, file management etc) - and by doing so, there was little recognition of the potential offered by digital literacy around empowerment, meaning making, scaffolding, communication, and building social networks. Take, for example, the opportunity to encourage, value and celebrate Freshman students' text creation (no matter what media they are using) - their identity, world view, culture, experiences and ideas - and how this might then be incrementally linked through activities such as reflection, to Higher Order thinking skills and research. Also, if words are most effective when they create images which in turn resonate with our emotions, the use of graphics, video and audio could offer opportunities to scaffold learners who are not fluent in print literacy, thereby supporting and embracing those learners who have previously been excluded from further and higher education.

I will, however, be studying the report to see if there are any recommendations around design, facilitation, assessment and evaluation that achieves the level of effectiveness and engagement indicated by the results of this study....