Fostering the Next Generation of Sustainability Professionals— Assessing Field Courses in a Sustainability Science Graduate Program

Sustainability science has been promoted actively both in research and education as a vibrant response to emerging sustainability challenges such as climate change, environmental degradation, food security, energy provision, and inequality. The main disciplinary foci of sustainability science are to understand the complex interactions between natural and social systems [1], [2],[3], [4], [5], and to create knowledge for sustainable development [6], [7], [8], [9]. Since challenges in sustainability generally require action to alter the status quo, the discipline’s approach is problem-based and solutionoriented [10], [11], [12], [13]. Moreover, interconnected problems [14] require researchers to go beyond their discipline of training. To reflect findings from research into actual practice, it is also necessary to cross the potential divide between academics and practitioners. Accordingly, sustainability science combines an interdisciplinary approach that employs academic knowledge from natural and social sciences to humanities, with a transdisciplinary approach that promotes co-design and co-creation of knowledge by diverse social stakeholders to address real-world sustainability challenges [15], [16], [17], [18].

While the research dimension of sustainability science has formed its own space and landscape within academia [10], [13], [19], [20], sustainability-related educational programs have also been promoted. According to a list from the journal Sustainability: Science, Practice, & Policy (SSPP)[21] there are more than 230 sustainability programs at the university level as of January 2016 [22]. Sustainability science education plays a key role in producing human resources with the literacy, knowledge, and skills required to actualize the recommendations of sustainability science research. Program curriculum and implementation must therefore reflect the interdisciplinary and transdisciplinary aspects of sustainability science. Students should be encouraged and trained to develop an interdisciplinary and transdisciplinary mindset, as the problems they address define the types of knowledge and methods necessary to propose possible solutions. Given the field’s problem-driven and solutionoriented approach, it is also critical to have linkages not only between research and stakeholders such as industry, government, and NGOs, but also between research and education for the continuous development of sustainability professionals. More collaborative and critical research approaches are necessary to guide social transformation towards sustainability [23].

Theoretical and applied literatures address the design of educational programs. The interdisciplinary approach of sustainability science brings together academic disciplines with diverging worldviews, and this in turn creates epistemological discussions. Such inter-paradigmatic collaboration and negotiation is considered a key characteristic of the field. In line with the epistemological discussion, the idea of ‘transepistemology’ [24] was introduced [24] to better describe the dynamic integration of different methods in sustainability research. According to Scheweizer-Ries and Perkins [24], trans-epistemology is the “cooperation between different personal knowledge systems” and society as a whole is the “ ‘producer’ of shared and socially constructed understanding of the world” [24]. The idea of sustainability is fundamentally normative and carries specific cultural values. It may also differ from person to person, so that sustainability science researchers must be able to imagine the diversity of views on a given topic and comprehend interlinkages between the viewpoints of different stakeholders. Accordingly, an educational program in sustainability must have the flexibility to accommodate awareness and tolerance of multiple epistemological views [25].

Regarding the design and operation of an educational program in sustainability science, Onuki and Mino introduced three key components: (i) knowledge and concept-oriented courses, (ii) experiential learning and skills-oriented courses, and (iii) transdisciplinary thesis research [26]. Mino and his colleagues later added the transboundary framework to emphasize the full range of scales, from the individual to the global, in order to examine subjects and problems from multiple angles [27].Tamura and Uegaki operate a sustainability science program in Ibaraki University, and raise another core concept for designing a sustainability science program, the “Mind-Skills-Knowledge” model of sustainability education [28]. In an analogy of sustainability science students with athletes who need to maintain their body, technique and spirit, the framework stresses a balance of different types of knowledge. Others have suggested declarative, procedural, effectiveness, and social knowledges, as well as their effective interaction [25].

In terms of the evaluation of sustainability science education programs, one challenge is to develop a method for investigating whether students are effectively acquiring the competencies required in order for them to actualize their knowledge as concrete actions for sustainability [28]. The work of Wiek and his colleagues provides a comprehensive discussion of five key competencies within a problem-solving framework [29]. While the proposed key competencies have been applied to assess students’ learning outcomes in a transdisciplinary course [30], a general need for research on pedagogy and evaluation in sustainability science programs remains.

To address this gap in the evaluation of sustainability science programs, this study aims to examine the problemdriven approach of sustainability science through student self-assessments of six field courses at the Graduate Program in Sustainability Science (GPSS-GLI), The University of Tokyo. Field courses in GPSS-GLI are designed for students to engage in collaborative research and to address real-world sustainability challenges in various topical cases. So far, field courses have covered countries in Africa (Kenya, Nigeria, and South Africa), Asia (China, Japan, Thailand), Europe (Denmark and Sweden) and Latin America (Costa Rica), and topics such as renewable energy, biodiversity conservation, and urban informal settlement [31].

These courses also aim to equip students with practical skills such as workshop facilitation, coordination with local resource persons, and field methodologies that can be applied to their thesis research. The general structure of each field course is designed by one or two faculty members who specialize on the given topic. One unit normally accompanies a cohort of four to ten students, and one doctoral student takes a leading role in the planning. Six field courses implemented during the academic year 2014–2015 are evaluated in this study (see Table 1 for an overview of the units). Four of these are Global Field Exercises and two Resilience Exercises, but these have equal weight and significance in the curriculum and are handled as identical in this analysis.

The assessment began with the development of a conceptual framework and methodology, implemented systematically in a subsequent phase. The first phase took place in the Tohoku Resilience Exercise, one of six workshops assessed in this study. While the exercise itself had an educational focus of having students grapple with the complexity of issues surrounding the regional reconstruction after the Tohoku Earthquake and Tsunami of March 2011, students simultaneously engaged in a reflective analysis that laid the foundation of this assessment effort [32].

This developmental phase began with a review and analysis of the conceptual framework of key competencies for sustainability science professionals [29] that had been used in a previous assessment of the said program [33]. The chosen framework was deemed appropriate for this assessment as a focus on real-world problems is characteristic of GPSS-GLI, and the five key competencies were identified for their relevance to sustainability science research and problem solving [29, 32]. Collectively, students reviewed this pre-existing framework and adapted the original definitions for use within an educational context [32]. The group then analyzed the linkages between each of the competencies and the activities and issues within the Resilience Exercise.

In order to hold pointed discussions about how different components of the field course-related activities contributed to participants’ personal development, there was a need to distinguish between active and passive learning, as well as of recognizing a competence as important in professions of sustainability science. As discussed by San Carlos and colleagues [33], a review of academic literature revealed a lack of consensus and clarity on the definitions of active and passive learning [34, 35]. For practical purposes, our understanding is that active learning involves active engagement of the students with the planned field exercise course activities. In other words, active learning is learning by doing, such as designing and conducting original field surveys and through firsthand interaction with stakeholders in the research topic. In contrast, passive learning is the unidirectional transmission of information through lectures and other methodologies that do not require active student engagement [35].

At the end of the week-long Exercise course with daily, reflective discussions, each student’s personal experiences were quantified for analysis using a questionnaire with the concepts discussed. This questionnaire was used throughout the subsequent assessment. The questionnaire assumed that the respondent would have received some explanation of the competencies prior to assessment, but listed definitions as shown in Table 2. Students were asked to rate the unit’s effectiveness in facilitating personal development of the respective competence beyond their baseline level. The assessment of each competence was threefold: for passive learning, for active learning, and for “recogni[tion]/agree[ment] about the importance of the competence for research on sustainability issues” (hereafter: “Recognition”). Responses were indicated according to a 5 point-Likert scale (1: very ineffective (no influence); 2: ineffective, 3: satisfactory, 4: effective, 5: very effective). Open space was provided at the end of the questionnaire with prompts encouraging comments on respondents’ personal experiences or feedback on the assessment itself.

All subsequent assessments were conducted after the completion of the field courses according to the following procedure. The authors contacted student participants of the respective course unit using e-mail and/or social media to schedule a course workshop. This correspondence involved all GPSS-GLI students who had participated in the course, with one exception where the student had already graduated and left the country.

Workshops were facilitated by at least one of the authors. A brief introduction of the assessment project was followed by inquiry about the unit’s educational and/or research objectives. Using a whiteboard or projected computer screen, students were then asked to list out the units’ activities. Next, the competence framework was introduced using the definitions on Table 2, and students were asked to identify linkages between the competencies and the listed activities. At the end, the questionnaire was handed out either electronically or on paper for students to complete individually. The total duration of the workshops averaged about 90 minutes, and all workshops were conducted between September and November of 2015.

Subsequent consultations with faculty and affiliated staff members supplement the above process as a means to consider the appropriateness of the completed assessment. To date, this process has consisted of an e-mail with a semistructured questionnaire to faculty and staff members associated with each field course. The e-mail included a summary of the student assessments for the respective unit, as well as cross-unit average scores. Another document outlined the intent of the assessment and prompted for responses as follows: 1) explanations and interpretations of the results; 2) reflections on the exercise design; 3) comments and feedback on the assessment itself. As some unit-specific comments would be traceable to individual faculty members, the document asked faculty members to indicate their willingness to have their comments attributable to the unit in question.

Field course units are referred to by the location: Minamata (Japan), Tohoku (Japan), Oasis (China), Costa Rica, Bangkok (Thailand), and Nairobi (Kenya). As shown in Table 1, the assessment had a high rate of participation, with full participation for four of six units. E-mail inquiries with faculty and staff members were followed up with reminders and reached a response rate of 86% (n = 7). As only one staff member was contacted, faculty and staff will hereafter be referred to as “respondents” to ensure confidentiality.

Figure 3 shows the questionnaire results. Columns (A) to (F) show results in each course unit by competence. Rows (1) to (5) show the results for each competence by course unit. Mean scores and standard deviations (SD) for each competence are shown by type of learning. The last column and row represent the aggregate means by competence and unit, respectively.

Figure 1 shows the mean scores for the five competencies by type of learning. Results indicate overall student satisfaction with the field courses, as all five competencies obtained scores higher than 3.0 (“satisfactory”) for all types of learning. The highest scoring competence was Interpersonal Competence (M = 4.17). The lowest scoring competence was Strategic Competence (M = 3.38).

High scores on Recognition indicate that students generally agreed with the literature on the relevance of the key competencies for sustainability science research [29]. Recognition scored higher than the other types of learning on four of five competencies (Anticipatory Competence (M = 3.89); Normative Competence (M = 3.89); Strategic Competence (M = 3.79); Interpersonal Competence (M = 4.27)). Regarding Systems Thinking Competence, Recognition only scored marginally below the mean score of 3.88 (M = 3.86).

Active Learning was evaluated more highly than Passive Learning for all competencies. This is intuitive, as the field courses are based on the concept of providing opportunities for active engagement in the field [36]. The difference was greatest for Interpersonal Competence, where the aggregate mean for Active Learning (M = 4.33) was 0.43 points greater than for Passive Learning (M = 3.90). In contrast, the gap between Passive (M = 3.17) and Active (M = 3.18) Learning was only 0.01 points for Strategic Competence.

Other notable results are the high scores on Interpersonal Competence (M = 4.17) and low scores on Strategic Competence (M = 3.38). Effectiveness with the development of Interpersonal Competence may be explained by GPSS-GLI students’ diversity in cultural, academic, and professional backgrounds as well as demographics [36]. Lower evaluations on Strategic Competence may be due to the expectation and desire of students to have a tangible impact on the study area, despite time and resource constraints that limit such impact in reality. Student and faculty respondents alike commented that courses focused on understanding past and current situations rather than on speculating the future. This is understandable given the one to two week duration of the courses and consistent with the interpretation regarding the lack of capacity of the courses to have tangible impact.

Figure 2 shows competence and mean scores for the six field courses assessed in this study. Mean scores by course unit were also higher than satisfactory (3.0). The highest scoring course unit was the Bangkok Unit (M = 3.85). The Tohoku Unit (M = 3.46) received the lowest scorings and had high inter-student variation in each competence, a result likely attributable to the extended and critical discussions unique to this unit [32].

Results showed varying tendencies across units on student assessments’ scores and standard deviations (Figure 3). Minamata Unit obtained high scores and low standard deviations for all competencies (see column (A)). Tohoku Unit yielded the lowest mean score (M = 3.46), with similar results excepting Strategic Competence (M = 2.75), which scored below “satisfactory”. However, standard deviations within each competence were high for all competencies and almost all types of learning (see column (B)). One respondent took particular note of the contrast between Minamata and Tohoku Units, as “both are designed as ‘experienceoriented’ [and with] similar concepts”.

The Oasis and Costa Rica Units were similar in their focus on research. However, evaluations by Oasis Unit students had large variation (e.g. standard deviations above 1.0 for Strategic Competence (Passive (SD = 1.30); Active (SD = 1.22); Recognition (SD = 1.58))). Students in both units were consistent in their high evaluations of the course’s impact on their Interpersonal Competence (Oasis (M = 4.87); Costa Rica (M = 4.20); see Figure 3, columns (C) and (D)). In particular, Oasis student evaluation for Interpersonal Competence was highest of all units (aggregate M = 4.87). These outcomes may be attributed to the emphasis on student leadership noted by the faculty respondents affiliated with the two units. One stated that this emphasis might have been interpreted as a lack of strategic vision in the design of the unit, explaining the lower evaluation on Strategic Competence.

General scoring patterns on the Bangkok and Africa Units are comparable. However, responses on the former unit had greater internal consistency (see Table 1, column (E), (F) and Figure 2). The Africa Unit yielded relatively high variation amongst students for Passive and Active Learning. An affiliated respondent observed that these relatively high variations might indicate that “the exercise led to variable experiences for different students”. Another respondent affiliated with the Bangkok Unit expressed surprise at the lower scores on Recognition. Regarding Systems Thinking Competence, this respondent suggested that more attention should be given to a “holistic view about the complex systems (economic, social etc.) relating to the environmental and health issues” addressed in the unit.

This self-assessment of field courses in GPSS-GLI addresses both the needs of the said program, as well as academic needs to assess the development of competencies necessary for sustainability professionals [29, 37]. Building upon the foundation of a previous assessment of GPSSGLI curricular activities conducted six years ago [33], the present study provides a more detailed and in-depth assessment of field courses, a core activity in the program.

The self-assessment method was generated in the previous exploration of GPSS-GLI student perspectives on curricular activities and the development of their competencies [33]. Student participation in the assessment and development of GPSS-GLI is consistent with the program’s emphasis on developing student leadership skills [36] and educational practices in which students can participate [38].

Responses on the validity of the assessment are mixed, yet overall positive. Most faculty members considered the competence framework to be an appropriate assessment framework for GPSS-GLI. Five out of the six faculty members consulted considered the results insightful to varying degrees. Comments included, “the results seem accurate,” and “results are convincing”.

Nonetheless, some were skeptical of the framework and/or fundamental approach of this assessment. One respondent considered the competence framework unfit for this assessment, another expressed that their understanding of the framework was insufficient to use it, and a third considered it necessary to differentiate between the two types of field courses (Global Field Exercise and Resilience Exercise) offered in the program. A further response questioned, “the overall assessment has to be looked at with a question mark”.

Indeed, limitations of this ongoing assessment must be carefully considered. First, students may have difficulties relating their field experiences to the development of their personal competencies. Moreover, the time between the field course and assessment varied from unit to unit. Notwithstanding the high rates of participation in the workshops (Table 1), the validity of our results must be interpreted according to the low numbers of participants per unit. Second, results depend on students’ comprehension of the competencies, and the relatively short workshops may have been insufficient to ensure adequate comprehension. One faculty member raised this issue and recommended incorporating an explanation of the competencies in the guidance before each field course.

Third, scores only reflect additional improvement of individual competencies that students considered attributable to the field courses. Accordingly, results are subject to variations in baseline levels. Individual experiences before, during and after the units play a great role in student assessment, and a respondent questioned “if [students] could really assess what outcome/impact they experienced for each key competence and by how much”.

Fourth, field courses were designed with unique objectives, none of which explicitly involved the said competencies. Nonetheless, one respondent interpreted that “assessment results show that this design was vindicated”, somewhat validating the methodology even if it differed from the original intentions.

A high or low score is not necessarily good or bad, but merely a reflection of the unit design. Results thus ought to be viewed in light of the respective unit. Alternatively, future assessments should consider incorporating or reflecting the unit design in its assessment framework so as to more appropriately assess field courses designed with varying objectives in mind. For example, intended objectives of the respective course unit may be integrated into the assessment framework to provide insights more relevant to the unit in question. However, condensing the main features of each unit design into the assessment framework would be extremely challenging. Instead, the authors believe that a post-assessment discussion with students and faculty could shed light on the results obtained and allow for an open discussion that would involve the units’ design.

Additional qualitative data on students’ experience could offer a deeper insight into the results and how the competencies were developed in each unit. One respondent suggested “one would have to have qualitative expressions about their experiences” in order to better analyze the assessment outcome.

Lozano has suggested that most of the tools available for assessing sustainability do not seem adequate for immediate application to the university setting. In general, responses to this situation fall under either 1) modification of the existing tools, or 2) creation of specific tools for universities [39]. The current assessment falls under the latter approach and attempted to cater to the characteristics of the field courses in GPSS-GLI. Any application of this competence-based assessment to other programs or universities should be conducted with care and upon fundamental reconsideration of the assessment approach. Within the program, faculty members must consider the appropriateness of the framework used in this assessment, as well as whether or not and how to incorporate its outcomes in the design of future course units.

While Wiek et al.’s framework was selected for its focus on sustainability science, Wiek and his colleagues specify that pedagogy was beyond the scope of their study [29]. Thus, the application of his framework to education is so far unique to this assessment project [32, 33], and the results must be interpreted with caution. For example, universal competencies other than those “key” to sustainability science have not been considered, and the list of the five key competencies so far identified has yet to be finalized [29]. The existence of other studies on sustainability in higher education suggest that attention should be also paid to competencies related to domains such as the affective learning outcomes of educational initiatives [40, 41]. Further, alternative approaches to assessment could be taken into consideration, such as the Integral Framework, employed by a GPSS-GLI faculty member in the design of one course unit [42, 43] but were beyond the scope of the present study.

More fundamentally, the objective and validity of an assessment need to be carefully examined. Most faculty members consulted in the assessment consider the development of an evaluation scheme for sustainability science education to be a necessary step in improving the program design. However, respondents shared the concern that an emphasis on assessment development may lead to program designs that excessively cater to evaluation. This concern is particularly relevant to the field courses, where, through direct exposure to the problems and through real-life interactions with residents of the field site, students’ learning outcomes extend beyond what was originally intended or anticipated. Field courses must thus maintain a certain degree of flexibility to encompass diverse forms of learning.

This study contributes to the development of a method to assess students’ learning outcomes of field courses in a sustainability science program. Through the case of six field courses in GPSS-GLI at The University of Tokyo, we address not only the development of the said program, but also academic needs to assess the key competencies for sustainability professionals. The results of the self-assessment suggest that the majority of field course participants felt satisfied with the knowledge and skills they acquired, and gained the ambition to further explore the respective topic areas. Students also recognized the importance of key competencies for sustainability professionals after participating in the field courses. Although the authors do not suggest that all courses be aligned with the key competencies, the study suggests that such alignment could raise students’ awareness of the competencies they are acquiring from the program they belong to.

We expect to tackle limitations of the self-assessment method of this study in future developments. In particular, the assessment framework may be altered to reflect the variety of field course designs. In terms of implementation, the framework’s concepts may be better standardized by building a common understanding of the methods and terminologies used across students and faculty. There should also be consensus within the graduate program on the role of the assessment and the appropriate level of effort dedicated to this task.

We believe that the results of this study showed enough evidence to support the usefulness and appropriateness of the deployed self-assessment. We consider the present study a successful and relevant step forward in the assessment of field exercise courses in the Graduate Program in Sustainability Science – Global Leadership Initiative of The University of Tokyo, and a contribution to the general development of mechanisms to assess key competencies for sustainability science research.