| Australian Journal of Educational Technology 2002, 18(2), 227-254. | AJET 18 |
Existing teaching-learning arrangements in many online courses may not offer sufficient scope for off campus and on campus students to work collaboratively. Tasks that support collaboration online, goal sharing and collaborative decision making provide computer programming students with experiential learning, replicating how they will work on completion of university study and when entering the professional world. This study describes the initial implementation and evaluation of an online environment designed to support a collaborative programming task. A holistic, context based approach to evaluating the success of the innovation is documented, together with the framework used for conducting the evaluation. The approach to evaluation is integrative and holistic, seeking multiple forms of evidence for collaboration, engagement and improved learning and teaching. In addition, the study outlines decisions that have to be faced by practitioners in supporting online collaborative skills and carrying through an evaluation of an initial implementation.
The benefits of collaborative learning are well-documented in the literature for learners at all levels and contexts (Johnson & Johnson, 2000) and research attests to the relevance and virtues of collaboration as a learning goal in its own right. For example, Slavin (1996) has demonstrated that collaborative learning has positive effects on motivation, social skills and attitudes. The collaborative approach is also extended by the idea of shared thinking and negotiation. McLoughlin & Oliver (1998) argue that participation in learning hinges on communication and dialogue between teacher and student or among peers. This communication serves both social and cognitive functions and assists learners to become part of a community of learners.
Collaborative learning has been demonstrated to bring about improved learning in content areas as it encompasses numerous attributes associated with effective learning, such as team problem solving and active, cooperative learning strategies (Bonk & King, 1998). For example, both high and low achievers in collaborative learning teams outperform their counterparts in individual learning environments (Webb, Troper & Fall, 1998). The social nature of collaborative techniques, the involvement and conversation among students have all been reported as positive learning processes (Ocker & Yaverbaum, 2001).
Several pedagogical strategies are referred to in the literature as ways to foster collaboration. The capacity to enable these processes has been recognised as a feature of successful web based environments in several recent studies (English & Yazdani, 1999). For example, Collis (1998) describes the adoption of group based project work wherein groups work together to share responsibility for solving complex tasks. In these groups, complex real world problems are shared giving students opportunities to articulate and defend their ideas and reach consensus on decisions. The learning processes are akin to social constructivist principles whereby articulation, peer review of ideas, revision and collaborative knowledge construction are fostered (Brush, 1998; Tenebaum et al, 2001).
Research indicates that the benefits of collaborative learning can be transferred to an electronic environment such as the World Wide Web. McConnell (2000) provides examples of groupware and computer supported collaborative learning systems to support communication, document sharing and asynchronous conferencing systems. Many course support systems, such as WebCT and TopClass, enable networking and conferencing within units of study, but are dependent for their success on appropriate pedagogical design that enables students to share their work (Koschmann, 1996).
There is widespread consensus that programming is a complex process involving a range of specialists working in a team environment to make decisions and create a product based on agreed specifications (Phillips, 1997). Similarly the work of Jonassen et al (1995) demonstrates the engaging of students as hypermedia designers in an instructional strategy that invites them to extend themselves and engage with others. As program designers, learners are encouraged to be creative, to integrate new knowledge and to pursue their own goals actively while engaging with others.
As shown by many studies within the multimedia industry, the process of producing programs and products from the inception of an idea to a finished product is a very complex task involving higher order thinking skills (Kennedy & McNaught, 1997; McLoughlin & O‘Sullivan, 2000). Programming tasks demand attention to both process and product outcomes, and learners typically enter a "culture of practice", or learning environment where they discuss and engage with the task as experts do (Collins, Brown & Newman, 1989). Confronting problems, sharing ideas and creating new knowledge are forms of authentic activity demanded by programming tasks. By way of comparison, the Extreme Programming (XP) approach has emerged in response to the dynamic demands of programming [http://extremeprogramming.org/when.html]. In this approach, the XP team consists of developers, managers and customers working together to deliver a product when it is needed and in the shape it is needed, utilising high levels of collaboration amongst development teams. In the context of the present study, XP was not an approach for novice programmers in their first year of tertiary study. Nevertheless, several comparable team approaches were built into task design.
Based on the literature review and the needs of students studying the unit, it was decided to use the online environment to create a qualitatively different learning experience to support collaboration and teamwork. The environment was intended to support and link both on campus and off campus students and to achieve outcomes that were neither affordable nor possible with existing technologies, such as audio conferencing or print. Thus, design issues are related not only to tasks and pedagogy but also to technical functionalities that enable communication and support individual and group collaboration. In translating successful pedagogical strategies for collaboration to an online environment, the following components were used to create processes and structures that were supportive of group work (see Figure 1):
Figure 1: Pedagogical elements of the online environment supporting collaboration
According to Wilson (1996: 5) learning environments are places where ‘learners work together and support each other as they use a variety of tools and information resources in their guided pursuit of learning goals‘. Both definitions are consistent with the elements depicted in Figure 1. In order to apply the findings of previous studies of successful collaborative learning to the context of the study, a holistic approach that linked design, implementation and evaluation was adopted.
The programming task was one of three assessment tasks that formed part of the overall assessment for the unit. Students were assigned marks for the task based on (i) successful task completion (ii) a team report showing planning and testing of the product (iii) participation in the bulletin board discussion tasks. This weighting was aligned with the intended learning outcome of collaboration and teamwork.
The task required students to create three C+++ classes that relied on each other according to set specifications. The three tasks were a schedule keeping class, a birthday class list, and a calendar class that integrated the other two classes in order to show which dates in a particular month had birthdays and which had appointments from the schedule. The tasks were straightforward and required basic programming skills, thus enabling students to focus on the team work components. Students had three weeks to complete the task from the time they were assigned the task to completion. All components of the task were submitted online.
Students were formed into groups of three by the course coordinator. Two of the three students were studying on campus, and the third was studying off campus. Primarily this arrangement was due to the weighting of students, to encourage online communication and to create a virtual classroom where off campus students could engage in collaborative tasks with peers.
| Cooperative feature | Pedagogical action |
| Orientate students to cooperative learning | Students were informed about learning outcomes, and the social processes |
| Plan for cooperation | Instructor names groups, assigns roles and provides resources |
| Create learning support | Define the task and provide scaffolding to ensure task completion |
| Monitor learning | Systematic observation and data collection on the interactions of groups |
| Evaluate collaboration | Group processes and learning outcomes are assessed |
| Allow self assessment | Participants provide feedback on the task and experience |
Collaboration was supported throughout the learning environment by undertaking a range of actions to ensure learner focus and orientation to the task, support and monitoring of learning, and an integrated form of assessment.
The design of tasks was informed by the research literature on the qualities of effective collaboration as identified by researchers of computer supported collaborative learning (CSCL) (Koschmann, 1996). The definition of team informing the research is that of Katzenbach and Smith (1993).
A team is a temporary or an ongoing task group whose members are charged with working together to identify problems, form a consensus about what should be done, and implement necessary actions in relation to a particular task area.This definition was used to orientate students to the notion of working collaboratively and they were informed that a high performance team of programmers would work towards a collective and agreed goal. Moreover, in addition to creating an environment conducive to teamwork, it was essential to create a task that enabled group members to display team skills. The task required the students to create programs that relied on each other according to set specifications. The intended learning outcome was not only the application of programming skills but also the creation of a team plan and product that exhibited teamwork, thinking skills and problem solving. The sub-tasks were interlinked and, in relation to other assessment tasks that were based on demonstration of programming skills alone, were designed to be simpler in order to allow a focus on the process of communicating with team mates and producing a collective result.
The task drew on team skills and content knowledge, and the underlying pedagogy was process based and directed at encouraging interdependence among learners, with a large measure of autonomy (McLoughlin & Luca, 2000b). As stated above, in order to support team skills, several support processes were provided to foster team building skills. Several phases and strategies were suggested to breakdown the task, encourage allocation of responsibility, maintain progress checks and foster team supporting behaviours.
Support for collaboration was achieved by designing an interface where scaffolding for collaborative communication and on task behaviour was built into the environment. That is, for each type of team communication desired, a corresponding category of posting was established. While posting a message, the interface required students to identify the purpose of their communication. This form of structuring was intended to prompt learners to reflect on the goal of communication. By specifying communicative goals that were supportive of teamwork, it was anticipated that students would complete the task as a team and achieve the intended learning outcome.
The scaffolds were intended to structure the task and increase team based interactions as follows:
Planning the team‘s time
Teams were comprised of students from different locations, lifestyles and with different time commitments. It was suggested the students be up front about their time and how much they could allow for the task over the three-week period. The students were also asked to suggest deadlines for completing the task.
Allocating responsibility within the team
With the three sub-tasks in mind, students were left the responsibility of dividing the task among the members of the team. Within each team there were differing perceptions of the difficulty of each sub-task and a range of skills. The task required planning of a team product.
Supporting (requesting support) team members
Once the students had been given some responsibility, it was time to go to work. Students were encouraged to offer assistance and advice to other team members, and to rely on their assistance when needed. Students were advised that seeking help from the unit‘s teachers should be a last resort, and should be planned as a communication that represented the whole team.
Stating progress
In order to keep team members informed, students were advised to state their progress in relation to the constraints that they had set themselves earlier.
Developing a testing strategy
In order to test the team‘s work, the students needed to collect their code in one place and compile it together. This required planning and negotiation. Students were also encouraged to anticipate how their code would be tested, to discuss the ambiguous aspects of the task and how they should improved the programming in order to create a reliable product.
Testing and redesign
Students were encouraged to discuss the success and failure of their testing, and prompted to determine changes required.
These supports for collaboration created task structuring and an expectation that all postings to the Bulletin Board would be focussed. While students posted their responses, there was no direct intervention from the teacher except when a direct response for assistance was received. Nevertheless, the tutors undertook a monitoring role to ensure that all students were able to log on, to offer assistance when needed and to detect when/if students who were supposed to be part of design teams did in fact log on.
| Phase | Focus | Application |
| Analysis and design | Learning environment analysis Design of the collaborative learning interface and task to foster teamwork | The learning environment is analysed to determine the shortfall in support for teamwork and to provide a rationale for the design of the innovation |
| Development | Functionality and accessibility to students Formative monitoring of the environment and learning processes | Throughout the learning task, student learning processes are monitored |
| Implementation | Summative evaluation of learning process and learning outcome was carried out | Evidence is sought on the nature of learning outcomes achieved |
This approach is similar in several ways to other approaches to evaluation, such as that used at the Open University, also categorised as holistic evaluation. Mason (1995) identified a number of critical factors that underpin successful evaluation, including drawing out central issues, or goals of the evaluation; using appropriate tools; contextualising technology use; and collecting data. These elements can be encapsulated into the Open University‘s framework for evaluation which focuses on three aspects: context, interactions and outcomes. These elements are described in Table 3 as applied to the present study.
Looking at the table by rows, the rationale provides an overview of why we need to evaluate the context, how learning interactions will inform judgements about student learning and how the learning outcomes need to be extended beyond purely cognitive concerns. The data row shows that different forms of data need to be considered, relating to the context, the interactions that occur because of the innovation and the learning outcomes.
Data relevant to redesign and improvement of task and the environment were produced, along with the recognition that there was a need to evaluate the innovation beyond the immediate context in which it was introduced.
| Context | Interactions | Outcomes | |
| Rationale | In order to evaluate the ICT innovation we need to know about the context of its use | Collecting process data on student use of the innovation helps us to assess what works and why | How to attribute learning outcomes to the innovation? Need to include cognitive and affective learning outcomes |
| Data | Designers‘ and course team aims | Online logs Analysis of transcripts | Changes in student attitudes and perceptions Need for triangulation |
| Methods | Curriculum analysis/ learning outcomes | Observation Content analysis | Interviews Questionnaires |
The specific evaluation questions were:
| No of postings by category | 20-30 Low (1L) | 30-50 Low (2L) | 50-60 Medium (1M) | 60-75 Medium (2M) | 75-100 High (1H) | 100-200 High (2H) | Total |
| No groups in each category | 4 | 12 | 6 | 6 | 4 | 3 | 35 |
Postings of each of the six groups were analysed according to the categories appearing in the dialogue, as structured by the Bulletin Board, which required students to identify each posting made according to the six categories (time planning, allocating responsibility, supporting team members, stating progress, developing a testing strategy, testing and redesigning). The results are displayed in Table 5 according to the percentage for each category. While each group used the structured Bulletin Board for postings, there were major differences in the pattern of postings that occurred. Table 5 shows that the two groups (1H and 2H) with the most postings also gave most attention to planning their team‘s time, allocating responsibility and supporting team members, and a smaller percentage of talk given to testing and redesign.
| Bulletin Board postings by percentage of total postings | 1L | 2L | 1M | 2M | 1H | 2H |
| Planning the team‘s time | 4 | 10 | 12 | 12 | 14 | 15 |
| Allocating responsibility | 15 | 15 | 16 | 18 | 18 | 19 |
| Supporting team members | 14 | 17 | 17 | 18 | 20 | 22 |
| Stating progress | 7 | 7 | 10 | 13 | 15 | 15 |
| Developing a testing strategy | 31 | 27 | 23 | 20 | 16 | 15 |
| Testing and redesign | 29 | 24 | 21 | 20 | 17 | 14 |
| Total | 100 | 100 | 100 | 100 | 100 | 100 |
Throughout the semester, groups remained fairly stable, though some members did not contribute much and some students failed to complete the task. Comparing the scores and types of interactions between teams, clear differences emerge in the number and function of postings between groups. Groups 1L and 2L had a significantly less amount of talk, and performed less successfully overall. The distribution of postings varied widely, with the 1L and 2L teams spending more time redesigning and testing, and the 1H and 2H teams spending more time on process skills such as supporting team members and planning time expenditure.
The results at the end of this initial stage of the evaluation show that the Bulletin Board did not scaffold successful collaborative learning for all groups, as groups 1L and 2L did not demonstrate effective communication processes.
It was concluded that further analysis of the postings was required in order to discover how the two most successful groups worked as teams, and how they differed from the other groups. This involved using another evaluation approach to gauge the extent to which the findings were valid and reliable. Emerging from this phase of the evaluation was the following question: was the high frequency of postings among the high achieving group (2H) the main factor that contributed to success, or was it the quality of interaction? Comparing the H groups with the L groups showed that the quantity of interaction differed, but there was not sufficient evidence to claim that the quality of interaction contributed to success. Triangulation of methods was therefore required.
Pattern 1: Goal clarification
A strong commitment to specific and measurable goals is considered to be the core ingredient of high performance teams and evidence of this was sought in the verbal protocols that occurred in Bulletin Board interactions. In this authentic assessment activity the extrinsic goal was set by the lecturer - marks for the assignment, but in addition one of the stated learning outcomes for students was improvement of team skills. Students were not explicitly asked to set a goal for the group (eg "We will all complete the assessment activity and receive full marks"). Four of the six groups analysed discussed a goal for the group (see Table 6). It seemed that in most instances the goal was assumed, as was the commitment of the group participants. In no instances was the goal questioned or interpreted. The members of the ‘high performance team‘ did begin their interaction to the ‘task‘ by referring to their success (or lack of it) in previous assessment activities. The following was one posting that reflected on team goals: "it might be easier this time with 3 heads working...but have to coordinate over e-mail so might take longer".
| Team process | 1L | 2L | 1M | 2M | 1H | 2H |
| Goals clarified | 0 | 0 | 1 | 2 | 3 | 3 |
Thus, at least one key ingredient of a high performance team was not well demonstrated by this group of students, who may have assumed that the goal was sufficiently clear, as it represented an assessment task as the external goal.
As with goal clarification, ground rules for group conduct never appeared in the discussions of team members. The rare occurrence was a humorous comment made by one group member to another inactive member.
No, you are not allowed to do that. People in this group have to be committed. No parties, no friends, no phone calls, certainly no early nights. Absolutely no leaving the computer until you have devised a class that does all that and more in one line.The task might have been achieved more easily if team members had acted in partnership with each other, and established how joint management was going to operate. As Katzenbach and Smith (1993: 242) remark ‘For a real team to form, there must be a team purpose that is distinctive and specific‘. While there were no indications of group conflict on the Bulletin Board, two teams disintegrated and failed to produce any outcome.
Pattern 2: Communication strategies clarified and maintained
For the highest performing teams, clarifying when and how they would work together was more significant than with the other groups. The type of hardware and software available for communication as well as the legitimacy of using other forms of contact were the focus of discussion among the low and medium groups. In contrast, the timing of communication and contact was the main topic of conversation for the highest performing teams. Several groups indicated the use of additional strategies like email to supplement their Bulletin Board communication. Only the highest performing teams actively used the Bulletin Board to share ideas and discuss the specifics of their work progress. Table 8 shows that the highest performing teams 1H, 2H and 2M used communication strategies and active participation to achieve task goals.
| Team process | 1L | 2L | 1M | 2M | 1H | 2H |
| Communication strategies clarified and maintained | 8 | 12 | 13 | 13 | 15 | 18 |
The need for effective communication did not go unnoticed by group members, with one student from group 2H stating: ‘I think the assignment is more about communication than the actual writing‘. The analysis showed quite clearly that process skills were considered quite important by the 2H team, who achieved highly in the task.
Pattern 3: Acknowledgment/reward and reflection
| Team process | 1L | 2L | 1M | 2M | 1H | 2H |
| Acknowledgment/reward for task completion | 4 | 4 | 16 | 17 | 17 | 18 |
| Support sought from other members | 15 | 15 | 19 | 18 | 19 | 20 |
| Reflection | 6 | 5 | 7 | 7 | 8 | 9 |
Analysis of transcripts showed that most teams used individual feedback between group members very effectively. From a simple ‘Received, Thanks‘ to a posting title of ‘Mission Accomplished‘, the M groups and H groups provided a significant amount of feedback to each other to enthuse and keep the group feeling positive about their progress, even though many problems were identified and worked out through Bulletin Board interaction (see Table 8).
The low achieving groups continued to soldier on towards the end of the task without actually expressing appreciation or satisfaction. Two brief postings in the form of ‘And yipee-yi-ay!!!!‘ and ‘Okay I have this message entitled "HOORAY HOORAY HOORAY" in my inbox. It looks pretty exciting I think I will stop posting messages and go and read it‘. The opportunity to use acknowledgment and verbal rewards was lost to this group. From group 2L came the comment:
Though honestly I still don‘t see this teamwork thing as being much more than helping each other. Is this what it is really like?Later in the course of events, the same person observed:
I can see now how the whole group thing comes into play. Though I think it would be a lot easier if I were sitting with you.One high posting group (2H) posted direct acknowledgments and ‘reward‘ statements when assistance was given or milestones achieved. This feature distinguished them from the L groups. Table 8 shows that the use of verbal encouragement and team support was frequent in the high performing and intermediate groups.
Reflective comments and self monitoring of process and task were more frequent in the high achieving teams, although instances were found in all groups. For example: ‘The assignments seem easy to do on the one hand, but when you start looking it is another story, we really do have to work out a strategy‘ and, in the two high performance teams, reflection on prior knowledge was evident in statements such as: ‘I was thinking that at least for each sub-task the idea is somewhat similar to the previous assignment, so we‘re not starting from scratch‘.
Pattern 4: Differences in group processing
The literature indicates that particular forms of supportive interaction, such as supporting group members and clarifying communication strategies, can be clustered as ‘group processing‘ (Tyson, 1998; Johnston & Johnston, 2000). Indicators of positive group interaction such as clarifying communication strategies, rewarding group members and seeking support are displayed in Tables 7 and 8. To obtain a composite score on positive group maintenance behaviours, the scores shown in Tables 7 and 8 were added to provide an overall ‘group processing‘ score. Table 9 shows the summary data for each of the six groups in terms of group processes. The table shows clearly that in the 1H and 2H groups approximately fifty per cent of interactions were devoted to group maintenance.
| 1L | 2L | 1M | 2M | 1H | 2H | |
| Group processing skills as a percentage of total talk | 23 | 26 | 45 | 55 | 59 | 65 |
Clearly, the results depicted in Table 9 show distinct patterns of difference in the quality of interaction in the six groups, with the high and medium teams showing better levels of group processing interaction than the low teams. These process results are later related to learning outcomes achieved.
Pattern 5: Individual effort and off task postings
There was some variation in the number of postings that were unrelated to the task, or categorised as social. There were more of these in the low achieving groups and were occasioned by group members not communicating on a regular basis and then using the Bulletin Board for chat unrelated to the task. For instance, many postings commented about workload in general or about living in college or the amount of downtime on the server.
Another interesting difference arising between groups was the number of postings given to individual contributions as opposed to collective and collaborative work. In these instances, individual team members would propose a solution to the entire task, without seeking input from others, and then simply circulate the solution. For example, this comment from an individual in one of the low achieving groups was not indicative of collaboration: ‘I have completed the birthday list if you want to take a look‘.
| Team | 1L | 2L | 1M | 2M | 1H | 2H |
| Individual contributions towards the project | 29 | 30 | 20 | 16 | 14 | 9 |
| Social and off task postings | 31 | 28 | 12 | 13 | 11 | 7 |
In contrast, this posting made by a member of 1H showed collaboration and help seeking behaviour: ‘Thanks for your help with the task Alan, it really helped me with a few little black spots I had in mine‘. Table 10 shows the percentage of social comments and off task talk by all groups.
Summary of patterns of successful collaboration
The patterns that emerged in the analysis are indicative of qualitatively different forms of interaction between members of the low achieving and high achieving groups. In depth analysis of intergroup interactions and group processing behaviours tended to corroborate the initial findings depicted in Table 5: that high achieving groups not only had more postings but also displayed qualitatively different interactions. However, additional insights were gained into how successful teams operated. Low achieving teams displayed fewer positive group processing interactions, such as supporting each other and planning, and consequently spent more time revising the final product and testing. The high achieving teams seemed to achieve a better outcome by spending more time supporting each other, planning and communicating. Clearly, for some students, teamwork was a new phenomenon, and the environment and task may not have provided sufficient support for this process based approach to learning. The final stage of the evaluation was to seek student views on the task and environment.
| % | 1L | 2L | 1M | 2M | 1H | 2H |
| Learning process Number of postings | 28 | 32 | 59 | 72 | 84 | 127 |
| Group process composite score | 23 | 26 | 45 | 55 | 59 | 65 |
| Learning outcome Group Project score (assessment) | 8 | 16 | 20 | 26 | 24 | 28 |
The evaluation of data therefore shows that the interactions in the M and H groups were frequent and productive, and resulted in high achievement in the task. On the contrary, interactions in the 1L and 2L groups were not productive and may have hindered task completion. Groups H and M not only demonstrated a higher number of productive collaborative interactions, but also achieved better outcomes. There is clearly a strong process-outcome relationship displayed in the data. The teams which displayed a strong commitment to maintaining communication, planning their time and building team rapport were more successful in the task completion. More of the discussion that took place in the H groups showed evidence of group collaboration and team effort, in contrast, L and M teams showed more individual contributions to the project. These patterns of interaction show quite clearly that in depth engagement in process skills contributed to successful task completion. In order to complete the evaluation, further data were sought from participants on their perceptions on the task and environment. This third analytical level completed the triangulation of data.
Figure 2: Students indicated that their team skills had improved
Does the task motivate learning and development of the desired skills?
When presented with the question ‘Do you feel the task motivated you to participate?, students responded unanimously in favour of the task, as shown in Figure 3. Some suggested that the task could be improved by providing additional support and resources.
Figure 3: Students indicated that the task motivated them to participate
Results from the assessment showed that the average mark for the assignment was a significant increase over results from previous assignments. A common problem with programming assignments is over confidence which leads to leaving commencement of an assessment task to the last minute, which leads to panic, incomplete submissions and often plagiarism. Students were aware that the task required time and planning. It was a refreshing change to see that most students commenced communication with their team mates almost immediately after being placed in teams.
Do the environment and learning task promote professional teamwork skills?
Most teams that participated communicated daily until the task was completed. Students were not placed in teams according to ability, but randomly in order to ensure that on campus students communicated with off campus students. Students‘ individual code submissions were assessed independently of the group product created among team members. The results of the analysis showed that most teams showed some teamwork skills, but the majority of students who were grouped in the 1L and 2L teams did not work effectively as a team, and did not show strong group processing behaviours.
Is the team working environment and task realistic to industry?
The students felt that the environment and the task were realistic in relation to what could be expected in industry. Students also reported that the experience had helped improve their programming abilities in terms of writing applications in separate files, compiling programs from separate files, testing programs and programming skills in general, which are all skills required in industry.
In summary, student perceptions of the task and environment were positive, but some students were over optimistic in reporting the successful achievement of team skills, as these are not supported by the results obtained (Table 5). Some students (about 20%) reported that the task could be improved by providing more support. This observation is discussed in relation to the overall evaluation.
How successful was the environment in supporting collaborative teamwork? (Process skills)
The scaffolds provided to students in the Bulletin Board assisted them in structuring their interactions and facilitated analysis of data. However, there were major differences between groups in the percentage of postings given to planning and allocating responsibility. The two groups with most postings (categorised as High and Medium) gave more time to planning, allocating responsibility and supporting team members than to testing and redesign. The groups with fewer postings spent a higher percentage of their time discussing strategies and redesign than the other groups, yet achieved lower marks for the task. This suggests that the more productive teams spent most time self managing communication and planning, and perhaps did not need to redesign the product. In terms of environment design, more emphasis on planning or scaffolds to assist students to manage their time and plan the project would have been desirable. The environment could have been improved by more direct intervention by the tutor to redirect and moderate discussion.
What evidence did the transcripts provide of successful collaboration and learning? (Formative and summative evaluation of the learning outcomes)
The transcripts of dialogue were again analysed in order to ascertain the degree to which effective collaboration was in evident in interactions of teams. A coding scheme was adapted from the characteristics of effective teams (Katzenbach & Smith, 1993; Johnston & Johnston, 2000) and the Bulletin Board interactions for this authentic assessment task were analysed. The results indicate a compelling relationship between group maintenance behaviours (communication, negotiation skills, reflection and goal setting) and effective task completion. Results showed that the quality of the completed program varied with aspects of the collaborative process that gave rise to it. For instance, team activity, positive group processing and frequency of posting was a positive indicator of successful task completion (Table 11). In addition, group processes scores showed that successful task completion was linked to a high frequency of verbal interactions that demonstrated concern for team members and group communication. While low posting groups appeared to be more task focussed in their conversations, they nevertheless scored poorly on the assessed program. Several factors may have impacted in the interaction processes within groups. It was clear that many students were unsure about how to work as a team, and may have felt frustrated having to depend on others for parts of the task solution. Further scaffolding of team skills and intervention by a tutor during the initial stages of the task would have been desirable.
What were students‘ perceptions of the task and environment? (Functionality and relevance to students)
The use of online technology in this unit was perceived by students to actualise team based learning in a number of ways:
In terms of learning outcomes, the evaluation revealed that further support for team processes would seem to be needed for distance learners to function effectively as teams. This could be achieved by task and assessment design and by creating more interdependencies. Collaboration needs to be an intrinsic part of the task initially set for students, and careful alignment of learning tasks, assessment and learning support needs to be considered.
In order to increase task realism, students need to be presented with some incentive to engage in dialogue with their team mates, such as the structured task environment presented in this study. If conflicts cannot be resolved within the team, the team should nominate a member who will formally approach an instructor and present a question. Instructors could be seen as consultants to the team, with advice coming at a cost. Another approach to would be to ensure that collaboration is inherent in the programming task.
The online environment could be improved by providing some additional facilities, for example, enabling intergroup comparison of results with discussion, a more immediate online communication medium (like chat) being available in the environment, and also some common area for storing and collaboratively testing the teams‘ files. It was also suggested that students have a choice of a general category for postings to the Bulletin Board. These suggestions have provided a rich source of data on the design features of effective online teams, together with an improved understanding of the importance of structured environments for students who are novices to virtual teamwork and peer partnerships.
Ongoing research needs to investigate additional scaffolds for team building skills such as goal setting, planning and positive group processing strategies in online environments. Evaluation of these processes and outcomes are likely to remain central to all learning environments, and learner centred evaluation approaches such as the Bain (1999) model enable design, implementation and evaluation to become part of an ongoing cycle of improvement.
Alexander, S. & McKenzie, J. (1998). An evaluation of information technology projects for university learning (CUTSD). Canberra: National Capital Printing.
Bain, J. (1999). Introduction: Learning centered evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.
Barnett, R. (1994). The limits of competence: Knowledge, higher education and society. Buckingham: SRHE and Open University Press.
Bonk, C. J. & King, K. S. (1998). Electronic collaborators: Learner centered technologies for literacy, apprenticeship and discourse. Mahwah NJ: Erlbaum
Brush, T. A. (1998). Embedding cooperative learning into the design of integrated learning systems: Rationale and guidelines. Educational Technology, Research and Development, 46(3), 1042-1629.
Collis, B. (1998). WWW-based environments for collaborative group work. Education and Information Technologies, 3, 231-245.
English, S. & Yazdani, M. (1999). Computer-supported cooperative learning in a virtual university. Journal of Computer Assisted Learning, 15(2), 2-13.
Extreme programming (XP). http://www.extremeprogramming.org/when.html [viewed Jun 2002, verified 25 Jul 2002]
Jegede, O., Fraser, B. & Fisher, D. (1995). The development and validation of a distance and open learning scale. Educational Technology, Research and Development, 43(1), 90-94.
Johnson, D. W. & Johnson, F. P. (2000). Joining together. Boston: Allyn & Bacon.
Johnson, D. W. & Johnson, R. T. (1996). Cooperation and the use of technology. In D. H. Jonassen (Ed), Handbook of research for educational communications and technology (pp. 1017-1044). New York: Simon & Schuster.
聯(lián)系客服
