Research & Evaluation

From AFSPedia

The Focus for Educational Research and Evaluation is on the Nature and Impact of AFS Programs.

AFS Long Term Impact Study

First Report Now Available - April 2008

Bettina Hansel, Ph.D. Project Director

Study Description/4 April 2008

Introduction

AFS’s mission is to enable people to develop the knowledge, skills and understanding to build relationships and interact effectively across cultures and, in so doing, create a more just and peaceful world. This is the most important thing we do as an organization. To understand the impact we have and to strengthen that impact it is critical that we research our programs and that we ensure that the organization as a whole has a solid understanding of the factors that promote intercultural learning and a capacity to use that understanding to improve the content and quality of our programs. In this way AFS can fulfill its mission, gain visibility as a leader in intercultural education, and ultimately better promote the programs we offer, particularly in the educational community.

In 2005, following the completion of a major study of AFS program participants by Dr. Mitchell R. Hammer, AFS realized the need for a longer-term perspective in understanding the impact of our programs. We chose to return to a group we had studied before, where we had seen significant short-term impact. In 1981-82, an earlier large-scale study of the impact of the AFS program was conducted with a group of US students who went abroad on both year-long and summer-long programs to any of 50 countries in the world. These former participants are now about 40 years old. This is also the age group that currently comprises a good portion of our host families and parents of the next generation of exchange students. It is also the age group that is typically at mid-career and that includes well-established community leaders. In short, this is the age where we should expect to see the real results of the individual’s intercultural development and their own impact on the community. We wanted to see how this group had developed, but also to expand to a worldwide sample of our program alumni of about this age.

Countries involved

With enough AFS organizations participating, it was possible to pool the costs and for the AFS network to fund this research with contributions from AFS partner organizations covering the major direct costs for survey development and data collection and with AFS International covering the overall organization of the study, the research design and data analysis. Participating in this study are: Argentina, Australia, Austria, Chile, Egypt, Germany, Indonesia, Italy, Japan, Malaysia, New Zealand, Norway, Switzerland, Turkey and the United States. All together, this group had contact information of some type for approximately 12,400 former participants, and all were committed to proceed with both staff time and direct financial contributions for the costs for the project.

Goals

Follow-up to 2002 study

AFS has often considered “life-long intercultural learning” to be an important outcome of the experience. Following the completion of the study by Mitch Hammer with the AFS participants of 2002, an experienced group of AFS staff and volunteers with a specific interest in education met in Colle, Italy to review the findings and determine the next steps for the organization. One of the recommendations of this group was that AFS launch a study of the long-term impact of the programs.

Hypothesis: Is the full impact related to DMIS “delayed”?

In thinking about our expectations for the growth of intercultural understanding among our participants and the actual levels of growth shown in the IDI measurements in Mitch Hammer’s study, we began to consider the hypothesis that a more substantial amount of time is needed to understand the AFS experience and give meaning to it. We also felt that the knowledge gained abroad needed time to be applied. Newly returned participants from the program will need more than 6-8 months to do this. This realization led the Colle group to its recommendation for further research.

Threshold Theory: One intercultural experience leads to more

Richard Brislin at one time proposed a threshold theory based on the observation that most people do not form any intercultural friendships, but that once they do form one good friendship across cultures, the threshold has been crossed and many intercultural friendships follow. Along this same line of thought, AFS believes that once a young person participates in the AFS program, they cross a threshold that leads them to seek out opportunities for increased intercultural contact. We wanted to test this theory in our long term impact studies.

The Web Survey and the Interview Study

Two parts to our research on long-term impact were then envisioned: one in which AFS would follow the same 2002 students with in-depth interviews (the Interview Study), and another (the Web Survey) that would survey the long-term outcomes of the AFS students from the 1980s. We chose returnees from this era since an earlier short-term study of impact had been done with students from 1981.

• The Interview Study requires substantial funding in order to organize the training of interviewers and ensure consistency in the reporting of the interviews across all nine countries. AFS began seeking funding for this project in 2006.

• The Web Survey, on the other hand was feasible if enough AFS organizations were able to pool funding. It was determined that this portion could begin as early as 2006.

Long Term Impact Study: Web Survey

AFS decided to open the research to AFS organizations in any country that had been active in the organization during the 1980s and that was ready to commit the necessary staff time and resources to ensure that the project could be carried out. Among the 15 AFS organizations who are participating in the study there were a total approximately 12,400 returnees with names and some contact information in the AFS databases.

In reviewing the 1981 survey, it became clear that this survey was not suitable either for web or for mid-career adults and could not be adapted without losing its validity. In contrast, the survey questions used in the 2002 Hammer study, including the IDI, were both adaptable or available in web versions and easily used or adapted for an older audience. AFS therefore decided to develop a new survey that included elements that could be directly compared with the 2002 survey.

We began with two general research questions:

• How are AFS alumni different than other adults?
• In terms of their impact on cultural understanding, how does the AFS experience compare with other experiences?

Given the volume of responses we expected, we realized that open-ended survey questions would need to be very limited. To have meaningful survey questions, then, our strategy was to begin with focus groups in most of the participating countries. These focus groups would include either AFS returnees from a similar era, or other adults that could be identified who had not gone abroad on a student exchange during high school. Focus group leaders were to focus on identifying a broad range of experiences, relationships, and background factors that the individuals in the group felt had had an influence on their attitudes and experience of other cultures. The results from these focus groups would be used to develop the web survey questions.

Focus Groups in 2006: Issues Identified

Focus groups took place in 12 countries in 2006. A larger report of the findings from these focus groups is also available. Below are some of the main findings from this effort.

International experience

• Experience living or visiting people in other countries
• Most AFS alumni and other adults have traveled abroad.
• Many people visit family members from other cultures.
• Study abroad experiences of various types are common among AFS alumni and other adults.
• Some people have international voluntary experiences that may have an impact on their attitudes about other cultures.

Relationships

• Many AFS alumni and others have a friend from a different cultural background
• Work and career frequently involve contact with other cultures for both AFS alumni and other adults.
• Some people have hosted an exchange student or someone else from another culture
• Some people have spouses or life partners from other cultural backgrounds

The community

• Most adults have some contact with immigrant groups or have neighbors from another culture.
• Many meet people from other cultures or develop new attitudes about other cultures through their involvement with their own children
• Some adults are involved in civic and community activities that bring them in contact with diverse cultures
• Everyone has the potential for media exposure to other cultures and Internet connections to people from other cultures.

Web survey with IDI in 2007

Using these findings, the following research questions were developed:

How do AFS alumni compare with other adults?

• Do AFS alumni have more experiences abroad generally than other adults?
• Do AFS alumni show greater intercultural sensitivity than other adults?
• Do AFS alumni become fluent in more languages than other adults?
• Are AFS alumni less fearful/more confident around other cultures than do other adults?
• Do AFS alumni have more extensive intercultural friendships and social and professional networks?
• Are AFS alumni more likely to choose careers that involve the need to interact with other cultures?

What factors influence one’s experience of other cultures?

• How important are values from parents in determining one’s own experience of other cultures?
• How important are childhood and student experiences in determining one’s own experience of other cultures?
• Does experience abroad relate to increased intercultural sensitivity?
• What is the relationship of confidence and fears related to personal safety to the level intercultural sensitivity?
• Do work experiences have an important impact in increasing cultural sensitivity?
• Do friends and social networks play a key role in intercultural sensitivity?
• Is there a relationship between intercultural sensitivity and the number of other languages spoken fluently?
• Are there differences between males and females in intercultural outlook or in intercultural experiences?

Life Directions

• Is there a relationship between one’s own intercultural sensitivity and one’s aspirations for his or her children?
• Is there a relationship between intercultural sensitivity and one’s preference for interculturally diverse communities?
• Is there a relationship between intercultural sensitivity and one’s career activities?

Current Status of Research

The first report of the study is now available. This is one of a series of reports planned for this project.

The AFS Long Term Impact Web Survey was launched at the end of April 2007. Each returnee was asked to provide the email address and a short message for two peers who had not participated in a secondary school exchange program. By mid-June, these emails were collected and a similar web survey was launched for this “control group” of nominated peers.

The survey period was extended to allow Chile's participation and to allow for additional time to gather data from a control group. The survey period is now closed (as of January 2008) with 1960 responding and 511 responses from nominated control group members. In November part of the team met in New York with Dr. Mitchell Hammer to make an initial review of the data collected up to that point and to design the plan for the full analysis. Since then, all data clean up has been completed and we are beginning with the data analysis using a phased approach.

Plan of Analysis

In the first phase of the analysis we focused on creating a profile of differences found between the AFS alumni and the control group. Findings are now posted on the AFS website.

In Phase 2 of the analysis, we will explore several factors that the initial data exploration suggested could be related to the developmental level of intercultural sensitivity and determine the extent to which each of these factors, including participation in the AFS exchange program, can be said to contribute to the intercultural developmental level. This will be a fairly long process of analysis and will include the aspect of life directions described above. We expect to complete this phase by September, 2008.

The third phase will provide greater depth to our understanding by focusing on the extreme groups: those who show the highest levels of intercultural development and those who show the least development on the Intercultural Development Inventory. We will also look in more depth at the factors identified in Phase 2 as having a major bearing on intercultural senstivity and their relationship to life choices and attitudes.

Finally, in 2009 we will single out two or three countries for in-depth, country-specific analysis.