A Case for Participatory Evaluation:
The PT3 Grant – MAGNETIC CONNECTIONS

Presented by

Mildred Murray-Ward, Ph.D.
Beverly Bryde, Ed.D.
Paul Gathercoal, Ph.D.
James Mahler, Ed.D.
Wendy Erlanger, M.A.
Veronica Virgen-Heim, M.A.
Carol Bartell, Ed.D.

California Lutheran University
School of Education

A paper presented at the annual meeting of the American Evaluation Association, November 9, 2002, Washington, DC

This research was funded through a U.S. Department of Education Preparing Tomorrow’s Teachers to Use Technology (PT3) Grant

Introduction

The purpose of this session is to describe the design and findings from the evaluation of the PT3 MAGNETIC CONNECTIONS grant conducted from fall 1999 to spring 2002. Baseline data and preliminary findings of the research on the project through December 2001 and reflections on the participatory evaluation design will be shared during this session.

Rationale

The evaluation design for MAGNETIC CONNECTIONS Project involved the practical participatory evaluation (P-PE) model, described by Cousins and Whitmore (1998). This model based on the more general participatory evaluation model (PE) defined by Rossi, Freeman, and Lipsay (1999) as:

An evaluation as a team project in which the evaluator and representatives of one or more stakeholder groups work collaboratively in developing the evaluation plan, conducting the evaluation, or disseminating and using the results.” (p. 36)

Stakeholders can include sponsors, managers, program developers and implementers, special interest groups, and/or program beneficiaries or targets. They must have a “stake” in the outcomes of the evaluation and possess enough program knowledge to contribute to the research process in meaningful ways (Green, 1986). Therefore, stakeholders (in this case the faculty) are subjects and researchers. The research they conduct generally focuses on professional practice. (Weiss, 1988 & Schmuck, 1997)

Participatory evaluation and the role of the stakeholder first appeared in the literature in Stake’s 1967 writing “The Countenance of Educational Evaluation,” in which Stake described his concern with the mechanistic models of evaluation that judged participants, rather than assisted them. Stake later expanded the notion of stakeholders in responsive evaluation, in which he suggested that evaluation must be sensitive to ways in which people build and use knowledge. However, the stakeholder was still not an active participant in the evaluation. Patton (1978) added his voice in calling for more practitioner involvement to enhance use of evaluation results. Later models offered increasingly complex roles for stakeholder participants (Worthen, Sanders & Fitzpatrick, 1997).

The use of participatory evaluation techniques is supported widely in the literature. Vanden Berk, Cassata, Moye, Yarabrough and Siddens (1999) described a strong parallel between organizational learning (a goal of participatory evaluation) and learning as described by dialectic constuctivist theories. In this set of constructivist theories, learners must create knowledge through interactions with other people and the environment. Such dialog is a major component in the PE as the evaluator and stakeholders complete their tasks. Evaluators tie participants’ prior experiences and skills to new ones associated with the evaluation. They also coach and support participant autonomy.

More specifically, the MAGNETIC CONNECTIONS Project involves Practical Participatory Evaluation (P-PE), described by Cousins and Whitmore (1998), in which participants as stakeholders and evaluators collaborate to create an evaluation plan, enhance its relevance and ownership, and finally increase its use in programmatic change. Burke (1998) describes the origins of PE has emerging from the social participation movements of the 1960s and 70s and the philosophical challenges of those such as Kurt Lewin, who promoted the idea of the practitioner as inquirer. P-PE contrasts with a second PE model called Transformative Participatory Evaluation (T-PE) which is designed to create sweeping social and cultural change, has its origins in the philosophy and research of those such as Freire, and is concerned with emancipating and empowering those persons who are powerless or socially disenfranchised.

Garaway (1995) and Pursley (1996) suggest that P-PE’s support the program being evaluated and assist with making decisions and solving problems. A number of models of P-PE have been used in the literature. In each one, the roles of the stakeholders have been altered. For example, Greene (1988) and Ayres (1987) reported evaluation models in which the stakeholders defined the problem, set the scope and interpreted data. Cousins and Earl (1995) described participation as including joint ownership and control of technical aspects of the evaluation.

There are two reasons to use PE (or P-PE) in lieu of more traditional evaluation models. First, Stake (1967) and Patton (1978), among others, suggest that PE impacts findings by keeping them relevant and useful to the stakeholders, and not just establishing program worth and utility. Cousins (1996) and Greene (1988) both suggest that involvement of stakeholders can increase utilization without sacrificing technical quality or credibility. Cousins and Whitmore (1998) explored a number of studies showing that such participation enhances utilization, because the stakeholders actually do the work. Furthermore, utilization can involve elements of planned change and impact organizational learning and change.

Second, PE is supposed to change organizations (Smith, 1999). Persons who participate the evaluation become more involved with the process, and in turn, see their organizations more clearly. This learning has been labeled organization learning (OL) and includes creating organizational memory and inserting new constructs and actions into the organization. Therefore, OL results in organizational change (Cousins & Earl, 1992; Robinson & Cousins, 1999). The stakeholders themselves benefit by gaining in feelings of self-worth, appreciation and acceptance of evaluation as a process, and the developments of skills of systematic inquiry (for example, see Whitmore, 1988).

Other positive impacts include those noted by Greene (1988), Cousins and Whitmore (1998), and Weiss (1988).

Ø      Keeps findings relevant

Ø      Maximizes potential for planned change

Ø      Develops appreciation & acceptance of evaluation process

Ø      Develops participants’ systemic inquiry skills

This approach to the evaluation helps to establish joint ownership of the evaluation and alters the participants’ roles to include the following:

1. Defining the evaluation problem
2. Setting the scope of the evaluation
3. Creating plan for evaluation activities
4. Assigning tasks
5. Analyzing the data
6. Interpreting the data      

Ayres (1987), Greene (1988), Weiss (1988), Reineke (1991)

PE does require certain conditions in the organizations, participants and evaluators, so that optimal results will most likely to occur.  First, the organization must value evaluation and provide necessary resources to complete the work. In addition, the institution must be committed to organizational learning (Cousins & Earl 1992).  The organizational culture and management must support the work (Poole, Davis, Reisman, & Nelson, 2001). PE works best in process or formative evaluations that involve installation or implementation of an innovation in a small or relatively restricted setting (Greene,1986). Finally, because PE requires collaboration and encourages pluralistic perspectives, the organization should be relatively free from conflict (Greene, 1986). 

Second, the participants must possess some important characteristics. As stated earlier, they must have a “stake” in the outcomes of the evaluation and possess enough program knowledge to contribute to the research process in meaningful ways (Green, 1986). They must also possess diverse perspectives (Mathie & Greene, 1997; Poole et al. 2001). If the participants do not have the necessary research training and experience, they must be willing and able to receive training and engage in the experience.  In addition, Turnbull (1999) has indicated that in order to maximize participation, practitioners must perceive their work place goals as participative. They must also believe that they have the freedom to participate to the degree they desire and can impact decision-making.Furthermore, they must believe that the PE process will helped achieve a desired outcome.

Finally, evaluators must possess, specific training in evaluation research skills. In addition, they must be accessible to the organization and be willing to assume the roles of teacher, coach, and mentor (Cousins & Earl, 1992; Robinson & Cousins, 1999; Brisolara, 1999; Fetterman, 1999). They must also be able to create a climate that respects personal perspectives and enables teams to work together (Robinson & Cousins, 1999).

Early use of PE in educational environments was advocated by Streich in 1986. Examples of teachers and administrators involved in PE may be found in Sick and Shapiro (1991), Green and Southard (1995), MacKay, Wellesley and Bazergan (1995), Duhon, Knight, Stallings, and Wiseman (1996), and Cousins (1996), among others. However, no studies involving participatory evaluations of higher education teacher training programs have been found.

Design

The evaluation of the California Lutheran University PT3 Project, MAGNETIC CONNECTIONS, features a formative/summative design recommended by Stevens, Lawrenz, Sharp, and Frechtling (1997) with a P-PE orientation described by Cousins and Whitmore (1998). The formative design explored five intermediate goals and the summative evaluation examined one outcome goal (Quiñones & Kirshstein, 1988).

Program Population and P-PE Sample

The design incorporates four separate groups who act as their own control. Higher Education Faculty are regularly appointed and clinical faculty of the School of Education who work in the Teacher Preparation Program and participate in the MAGNETIC CONNECTIONS PROJECT. The K-12 teachers are a purposeful sample of teachers from each of our partner schools who have participated in co-teaching, professional development, and supervision of student teachers. Pre-service teachers represent a purposeful sample of pre-service teachers from all three types of school settings served by CLU.

Participants in the P-PE - The participants are primarily faculty of the School of Education who work in the regular and Special Education Teacher Program. All faculty have had some research training, but few had training in program evaluation. Some have been more involved in research than others. The amount and level of participation varied in the group. The most frequently involved were the director and associate director who acted as internal evaluator. They participated in the design and development of the evaluation plan and all training activities. The associate director co-designed the instruments and training activities and helped deliver the training.The two clinical faculty members were the next most active. They attended all training and participated in sessions on coding and conducted focus groups with students. In addition, they created several of their own studies that were “spinoffs” of the technology training. For example, one clinical faculty member completed a study on tutoring via TappedIn and another completed work on conducting parent meetings on line.

Training in Evaluation - The seven training sessions were conducted over two years and were open to all School of Education faculty members. Topics included an overview of evaluation, SPSS, designing focus group protocols and surveys, document analysis, interview coding, general qualitative data analysis, review of pre-data for analysis, publishing results, and development of classroom observation forms and rubrics for evaluation of the electronic portfolio student work. Following the recommendations of Morin (1985), the faculty members’ roles were open and negotiated. The input of the faculty in discussion and design was welcomed and used. In addition, they were paid for their work on the evaluation and training in research techniques.

Data Collection/Instrumentation

A total of six different techniques and instruments are being employed in the evaluation. These techniques and instruments involve both quantitative and qualitative data. All goals are addressed by at least two instruments, allowing for triangulation of findings. All instruments have been selected or created to produce the most valid and reliable results.

A summary of target groups and data collection is located in Table 1.

Document Analysis – The document analysis is being completed on the Teacher Preparation Program course syllabi (pre and posts analysis), portfolio structure, and professional development training materials. The syllabi were pre-analyzed using the MAGNETIC CONNECTIONS Syllabus Analysis Form created by the director, co-director/internal evaluator and external evaluator.

Pre/post Technology Use and Knowledge Self-Analysis – The instruments are completed by selected higher education faculty and K-12 and pre-service teachers at the beginning and completion of the project. The instruments are adaptations of the MAGNETIC CONNECTIONS Self- evaluation Rubrics for Advanced Teacher Computer Use Student/Teacher Technology Survey. All are based on the instruments, Self- evaluation Rubrics for Basic and Advanced Teacher Computer Use, and Self-evaluation Rubrics for Teacher Internet Use, created by Quiñones and Kirshstein (1988).

Examination of Portfolios - The pre-service teacher portfolios will be examined using the Pre-service Teacher Electronic Portfolio Rubric checklist for contents and a qualitative coding structure for narrative contents such as a reflection paper. Faculty are developing the checklist and qualitative coding structure this spring.

Observation of faculty and pre-service teachers – The higher education faculty and pre-service teachers will be observed during their work in university and school classrooms. The observations are to be videotaped for knowledge and use of technology and coded using the MAGNETIC CONNECTIONS Observation Form, developed by the School of Education faculty.

Focus groups – A series focus groups with higher education faculty and pre-service teachers have been conducted starting with the spring of 1999. The focus group instruments titled the MAGNETIC CONNECTIONS Faculty Focus Group Protocol and Pre-service Teacher Focus Group Interview Protocol were designed by the project director, co-director/internal evaluator, clinical faculty and external evaluator.

Analysis of online seminar transcripts – The online or distance learning seminars held in TappedIn have been collected and are being analyzed for types of student/faculty interaction and knowledge and use of technology.

Training of the higher education faculty on data collection techniques, instrument design, and data analysis further assured collection of the most objective and reliable data possible. All forms and instruments were field tested and revised before actual use.

Table 1

Target Groups and data Collection Techniques

The data generated from the evaluation are quantitative and qualitative in nature. The quantitative data from the pre/post self-analysis of the pre-service teachers and higher education faculty are being explored for differences in responses from the start to the end of the project. The qualitative data collected from the document analysis, focus groups, electronic portfolio analysis, focus groups, observations and distance learning transcripts are being analyzed. As with other events in the evaluation, the faculty play key roles in the data analysis. They are completing the actual analysis and discuss the findings to make ongoing corrections and improvements to the program.

Results

The results from the evaluation have been grouped into three categories: 1. Program developments, and 2. Faculty Outcomes and 3. Pre-service teacher (student) Outcomes.  The results presented in this paper represent those from 1999-2000, 2000-2001 and the fall semester of 2001-2002. A number of changes have been made in the Teacher Preparation Program, and are listed in Table 2 by academic year. The changes in faculty have been located in Table 3 and pre-service teacher changes have been noted in Table 4.

Program Developments –

Table 2 shows that the Program has steadily increased the use of technology among the faculty and students. First technologies added to the faculty and student use of word processing and library searches included TappedIn, the electronic portfolio. In 1999-2000, Clearphone was introduced. In 2000-2001, the syllabi were created to conform to California technology standards. In fall 2000, the electronic portfolio was piloted with 80 students, expanded to another 80 in spring 2001, and finally required for all students in fall of 2001. During 2000-2001, technology introductions included WYSIWYG, scanning and digital cameras, Quicktime movies and the use of Palm Pilots.

In Process for 2001-2002. During the 2001-2002 year, a number of activities are planned. First, all of the partner schools were connected to CLU. Clearphone is now completely installed and being used to supervise pre-service teachers at the partner schools. In addition, final faculty and pre-service teacher focus groups will be conducted. Third, the final syllabus analysis will be completed. Fourth, post pre-service teacher technology use surveys will be completed and compared with the previous years’ results. Finally, the analysis of the qualitative data, including on-line transcripts and school classroom observation data, will be completed.

Faculty Developments –

Over the course of the grant, the faculty have developd their technology skills and added training and participation in evaluation methodologes. While initially feeling moderately comfortable with traditional uses of technology and Internet/WWW and email basics, the faculty were less comfortable with more sophisticated uses such as chats, real time and push technologies, web page construction, netiquette and ethics. Much of the technology instruction has focused on these tools and a focus group conducted at the end of the second project year revealed much growth and more comfort with these technologies.

In the area of research and evaluation, the faculty have demonstrated the phenomenon described by Cousins & Earl (1992) in which practitioners who were trained in social science research methods were more inclined to use and create their own research.

To date, the faculty have developed more skills, have initiated their own studies, and have been active in delivering papers at conferences. Faculty participation in the evaluation itself has centered on skill development, instrument construction, and analysis of much of the pre-data.

In Process for 2001-2002.

The faculty have more conference papers accepted for this coming year and a number have published or are working on peer-reviewed journal articles, as well. This summer and the coming year will be spent on analysis of post program data.

Pre-service Teacher Developments –

At the start of the project, most pre-service teachers possessed computers or had ready access to them. However, most felt comfortable only with personal uses and search tools. Few had training or experience with instructional applications, except drill and practice activities. In the initial focus group, the pre-service teachers saw the benefits of technology, but felt that it was difficult to learn and logistically too complex. In addition, they felt that faculty did not provide quick feedback on their efforts.

By the end of year two, the faculty noticed that the pre-service teachers’ fears seemed to have diminished, to be replaced by concerns about technology use in their own and their students’ learning. Logistically, the electronic portfolio was easier to use with the introduction of WYSIWYG.

In Process for 2001-2002.

At this point in time, all pre-service teachers use the electronic portfolio and TappedIn is used in all seminars. Some students have also used TappedIn in their program exit interviews. Clearphone, now available in our partner schools, is being used as a tool for observation. Current pre-service teachers now receive observation feedback collected on the Palm Pilots and sent via email.

Conclusions from the Study

The study resulted in several tentative conclusions. First, technology use increased dramatically in the 2 1/2 years and it increased in non-traditional ways that included:

• Seminars conducted on-line
• Electronic portfolio used to evaluate students
• Focus groups conducted on line
• TappedIn used to project parts of portfolios to students to reflect on contents
• Supervisors using Palm Pilots to record field work evaluations

Second, faculty have grown in their skills in using technology and conducting research. The evidence of research productivity is quite strong. In the future, faculty will also be asked to work on more data analysis.

Conclusions from Participatory Evaluation Methodology – Pluses

Participatory Evaluation enabled the faculty and project management to respond with speed after being informed about the project implementation processes and practices.  Information generated from the evaluation could be used to redirect project activities to achieve desired results and progress toward the project vision.  For example, data collected from preservice teacher and faculty focus group interviews could be applied immediately to the use of technology in the Teacher Preparation Program.  As we were informed that Tapped In was best used when the seminar leader provided students with an agenda and kept seminar participants on task by following the agenda and ensuring that all items were addressed in a timely and professional manner throughout the online seminar; this strategy, brought to light through focus group interviews, was immediately shared among all faculty who attended Tech. Monday workshops, and the use of Tapped In was enhanced and modified to benefit preservice teachers’ and faculty’s educational needs.

One of the goals of the grant was to redesign the Teacher Preparation program to meet the new California standards for teaching with technology by infusing them throughout the program.   Instruments used by stakeholders in the Participatory Evaluation process were designed to evaluate how well the project was meeting this goal and to provide information about “holes” in the curriculum that needed to be addressed.  As a result, modifications in project implementation were focused around the use of technology in teacher preparation and participatory evaluation enabled faculty to respond quickly by designing curricular activities that taught and tested preservice teacher candidate’s skills and conceptual development according to the state standard for teaching with technology.

Participatory Evaluation meant that faculty needed to learn how to use various technological tools for making meaning of the data collected.  Evaluation Fridays, were established and conducted by the external and internal evaluators and addressed such topics as developing evaluation instruments and using SPSS to analyze data.  As a result, faculty who participated in these “Evaluation Fridays” are able to better design instruments to investigate research questions and use technology tools to analyze the data they collect. Faculty involved in

Participatory Evaluation have presented their findings at International, National, Regional, State, and Local Conferences, including the American Educational Research Association (AERA), American Evaluators Association (AEA), and American Association for Colleges of Teacher Education (AACTE).  They have published findings in peer-reviewed publications, including Educause Quarterly, and they have contributed many scholarly papers registered in the ERIC Clearinghouse database.  Participatory Evaluation has increased the scholarly productivity of faculty in the School of Education and enhanced their reputation as teacher/scholars.  This will advantage them as they become eligible for promotion and tenure.

More questions are being generated as faculty participate in the evaluation process.  As a result, “spin-off” studies develop and faculty pursue questions that are relevant to the use of technology in education, but do not directly tie-in with the mission of grant activities.  For example, one faculty member used Clear-Phone to tutor K-12 students at a distance and compared affective and effective indicators to see if there was any significant difference.  Her design was quasi-experimental in nature and included a pretest, treatment, and posttest of preservice teachers who conducted the tutoring.  She found that there were no significant differences in the affect and effect of tutoring at a distance through Interned video-conferencing technologies and she presented these findings at the AERA Annual Meeting in 2001 and published her paper in the ERIC Clearinghouse.

In summary, the advantages associated with Participatory Evaluation are:

Ø      Faculty use findings to modify the program and instruction practices.

Ø      Changes are more focused on technology and new CA standards.

Ø      More faculty are using more technology in research.

Ø      Faculty have developed their inquiry skills.

Ø      Faculty have presented papers for International and National conferences.

Ø      3 new “spin-off" studies on technology use.

Conclusions from Participatory Evaluation Methodology –Challenges

There is an old saying, “If you go alone, you can leave when you are ready; if you go together, you must wait for everyone.”  Such is Participatory Evaluation, you cannot evaluate until everyone is ready.  The need to train faculty in evaluation techniques has necessitated the ever-increasing expansion of the delivery timeline.  The data collection has been timely, but the data analysis is slow.  Faculty always find ways to fill their time, and unless they have a compelling reason to know, they will find other ways to spend free moments. 

The process is more labor intensive than if a single evaluator took responsibility for conducting the evaluation.  Faculty need to be educated, they need to help design the evaluation instruments, they need to use the instruments to collect data, they need to learning how to use the technological tools and then analyze the data, then they need to synthesize the analysis and make recommendations to the stakeholders.  Even though faculty are paid for their time, the process is incredibly labor intensive and many faculty simply refuse to be involved in all aspects of the evaluation.

Participatory Evaluation by definition means many evaluators and the management of data has been a challenge.  Since so many individuals are collecting and analyzing data, the sharing of these analyses and a communal collecting point has been an ever-present challenge for the internal and external evaluators who oversee the process. 

The initial design of the evaluation has received much lauding from federal auditors.  However, maintaining the methodological rigor inherent in the design has been a challenge for all.  Limitations in project resources (the grant was not fully funded), faculty time and training, and faculty beliefs that teaching with technology is important, have all contributed to sacrifices in rigor. 

Prior to developing the concept of Participatory Evaluation, faculty raised the question, “What proof do we have that this technology is enhancing the teaching and learning of our students?”  This question was addressed at the first technology-training meeting of faculty and information regarding the benefits of the infusion of technology into teaching and learning was provided.  To obtain faculty participation, it must be demonstrated to faculty that their involvement has payoff for them and potentially dramatic payoff for the academic unit.  Faculty beliefs need to be addressed first and they need to be addressed often and again.  We have found that the implementation process cannot address faculty beliefs once and think that it is finished.  Faculty will question the use of technology in teaching and evaluation every step of the way.  Project leaders must be knowledgeable of reasons why technology is good for education and how it works in the best interests of students. 

In summary, the challenges associated with Participatory Evaluation are:

Ø      The evaluation process has been slowed down because of the need to train faculty and accommodate faculty time constraints.

Ø      The process is incredibly labor intensive.

Ø      Data management is a major  “challenge”

Ø      Some methodological  rigor was sacrificed.

Ø      Faculty must constantly be convinced that technology can make a difference.

Other questions to be explored in the effectiveness of the P-PE:

As with all evaluation, more questions emerge as the evaluation progresses.  These questions seem to emerge in response to our experiences with Participatory Evaluation:

Ø      What is the best way to train participants? (Cousins and Earl, 1992)

Ø      What have the participants learned? What research has been generated on their own? (Brisolara, 1998)

Ø      How have they used their new knowledge (depth of participation during and after study)? (Cousins and Earl, 1992)

Ø      Have we improved the quality program or services?  (Brisolara, 1998)

Ø      Has authentic participation in the program increased? (Brisolara, 1998)

Table 2:  Program Developments

Table 3:   Faculty Outcomes

Table 4:   Pre-service Teacher Outcomes

References

Ayres. T. D.  (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program Planning, 10, 263-271.

Burke, B. (1998). Evaluation for change: Reflections on participatory methodology.

New Directions for Evaluation, no. 80, 43-56.

Cousins, J. B. (1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22(1), 3-27.

Cousins, J. B. & Earl, L. M. (1992). The case for participatory evaluation. Educational  Evaluation and Policy Analysis, 14(4), 397-418.

Cousins, J. B. & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, no. 80, 5-23.

Duhon, K, Knight, S. L., Stallings, J. A., & Wiseman, D. (1996). Evaluation of complex programs using participatory evaluation. A paper presented at the annual meeting of the American Educational Research Association, New York. (ERIC Document Reproduction Service No. ED 405353).

Fetterman, D. M. (1999). Reflections on empowerment evaluation: Learning from experience. The Canadian Journal of Program Evaluation, Special Issue, 5-37.

Garaway, G. B. (1995). Participatory evaluation. Studies in Educational Evaluation, 21(1), 85-102.

Green, J. & Southard, M. (1995). Participatory approaches to evaluation for supporting school management: Three case studies. A paper presented at the annual meeting of the American Educational Research Association, San Francisco.  (ERIC Document Reproduction Service No. ED 384634).

Greene, J. G. (1986). Participatory evaluation and the evaluation of social programs: lessons learned from the field. A paper presented at the annual meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. ED 291766).

Greene, J. G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12(2), 91-116.

Mackay, R., Wellesley, S, & Bazergan, E. (1995). Participatory evaluation. ELT Journal, 49(4), 308-317.

Mathie, A. & Greene, J. C. (1997). Stakeholder participation in evaluation: How important is diversity? Evaluation and Program Planning, 20(3), 279-285.

Morin, A. (1985). New trends for new needs in instructional technology theory and development. A paper presented at the annual meeting of the Association for Educational Communications and Technology, Anaheim, CA. (ERIC Document Reproduction Service No. ED 256327).

Patton, M. Q. (1978). Utilization focused evaluation. Beverly Hills, CA: Sage Publications.

Poole, D. L, Davis, J. K., Resiman, J, & Nelson, J. E. (2001). Improving the quality of outcome evaluation plans.  Nonprofit Management and Leadership, 11(4), 405-421.

Pursley, L.A. (1996). Empowerment and utilization through participatory evaluation. Unpublished doctoral dissertation. Department of Human Services, Cornell University.

Quiñones, S. & Kirshstein, R. 91998). An educator’s guide to evaluating the use of technology in schools and classrooms. Washington, DC: U.S. Department of

Education. Available: http://www.ed.gov/pubs/EdTechGuide/title.html

Reinke, R.A. (1991). Stakeholder involvement in evaluation: Suggestions for practice. Evaluation Practice, 12(1), 39-44.

Robinson, T. T. & Cousins, J. B. (1999). Internal participatory evaluation as an organizational learning system: A longitudinal case study. A paper presented at the annual meeting of the American Evaluation Association, Orlando, FL.

Rossi, P. H., Freeman, H. E, & Lipsay, M. W. (1999). Evaluation: A systematic approach (6^th edition). Thousand Oaks, CA: Sage.

Schmuck, R. A. (1997). Practical action research for change. Arlington Heights, IL: IRI Light Training and Publishing.

Sick, P. & Shapiro, J. P. (1991). Utilizing participatory evaluation in a public school setting: An assessment of teacher involvement in decision making. A paper presented at the annual meeting of the American Educational Research Association, Chicago. (ERIC Document Reproduction Service No. ED 355771).

Smith, N. (1999). A framework for characterizing the practice of evaluation, with application to empowerment evaluation. The Canadian Journal of Program Evaluation, Special Issue, 39-68.

Stake, R. E. (1967). The countenance of evaluation. Teachers College Record, 68, 523-540.

Stevens, F., Lawrenz, F., Sharp, L., & Frechtling, J. (ed.) (1997). User-friendly handbook for project evaluation: Science, mathematics, engineering, and technology education. Washington, DC: National Science Foundation.

Streich, W. H. (1986). Public school curriculum improvement through participatory evaluation. A paper presented at the annual meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. ED 274080).

Turnbull, B. (1999). The mediating effect of participation on evaluation use. Evaluation and Program Planning, 22, 131-140.

Vanden Berk, E. J., Cassata, J. C., Moye, M. J., Yarbrough, D. B., & Siddens, S. K. (1999). Teaching and earning: Highlighting the parallels between education and participatory evaluation. A paper presented at the annual meeting of the American Evaluation Association, Orlando, FL.

Weiss, C. H. (1988). Evaluation (2^nd edition). Upper Saddle River, NJ: Prentice Hall.

Whitmore, E. (188). Evaluation as empowerment and the evaluator as enabler.   (ERIC Document Reproduction Service No. ED 294877).

Worthen, B. R., Sanders, J. R., & Fitzpatrick, J.L. (1997). Program evaluation: Alternative approaches and practical guidelines (2^nd edition). New York: Longman.