How Do They Know They Know:
Evaluating Adult Learning

Vella, Berardinelli, Burrow
1998, Jossey-Bass Inc., San Francisco
(authors are affiliated with the
Jubilee Popular Education Center, Inc.
Raleigh, NC)

Preface:  A nice summary of non-formal education:

1. Participation of the learners in determining what is to be learned through needs assessment
2. Dialogue between learner and teacher and among learners
3. Small-group work to engage learners
4. Visual support and psychomotor involvement
5. Accountability: “How do they know what they know”
6. Participative feedback on results of programs
7. Respect for learners and teachers
8. A listening attitude on the part of teachers and resource people
9. Learners do what they are learning

Chapter 1:  “A New Way of Thinking About Evaluation”

“Utilization:” How can you measure the learner’s continuing use of a new skill, new concept or newly developed attitude? Needed: a course planning and designing process that anticipates certain results. At Jubilee this involves:

1. Prior to the course, get relevant baseline data from each upcoming participant (to “inform” not “form” the course)
   1. Picture: a 30 min. videotape of participant actual teaching
   2. Preparation: Evidence of how participant prepares for a presentation. Guidelines? Steps?
   3. Program: An actual program plan participant used in the past
   4. Perception: Have a former student(s) answer (in writing): 1) What do you perceive are the strengths of—–as demonstrated by this experience? 2) What aspects would you like to see him or her work on and improve?
2. At Jubilee, the trainers call up-coming participants to discuss expectations, fears, etc.
3. Have learners begin building a learning portfolio so they can see their own progress. Enable to compare new designs, skills, and attitudes, with the old.

Chapter 2:  “Building from the Base”

Characteristics of Effective Evaluation:

1. Must be objective
2. Must identify the important elements of an education program
3. Must match organizational philosophy
4. Must be identifiable and accessible (process must not be too difficult and time-consuming)
5. Must focus both on outcomes and the process. 1) Did we accomplish our objectives? 2) Did we accomplish them in an effective and efficient way? (My note: #2 reflects a N.A. value. A more Biblical question might be: “Did we accomplish them in a way that honors God? Another important question: “What did I learn as a trainer? How can I be more effective next time based on what I learned here?”)
6. Must be integrated into the educational planning process, otherwise it will be ignored, be too time consuming and expensive, and/or be less objective.
7. Evaluation must be “owned” by trainers and learners.

Evaluation Axioms:

1. Evaluation does not just happen…must be carefully planned, just as the training program is.
2. Evaluation must be done by experts. Learners have a critical role to play in terms of evaluation, but they cannot speak wisely to all aspects of the training (e.g. future effectiveness of the training). Other trainers/outside observers are needed.
3. Effective evaluation returns more than it costs
4. Evaluation can be accomplished in many ways

Evaluation must be owned by both the trainers and the learners, otherwise it will be viewed as an outside imposition and therefore won’t be supported.

Key Questions to Ask when Developing an Evaluation Plan:

1. What is the purpose of the evaluation? (KISS)
2. What should be evaluated? Align evaluation with what is being taught.
3. What are the sources of evaluation information?
4. What are the methods for gathering the information?
5. When should evaluation be completed?

Program Planning Process:

1. Purpose of education program
2. Learner SKAs (skills, knowledge, attitudes) to be developed
3. Education program design decisions
4. RESULTS:  Learning that occurs—changes that occur in SKAs as a result of the program
5. RESULTS:  Changes in performance (“transfer”)—learning from the program that is applied in the learner’s work after completing the education/training program
6. RESULTS:  Impact (organizational improvement as a result of the learner’s work)

A comprehensive evaluation process should allow for the measurement of all 3 types of results (#4-6 above)

1. Learning: immediate and specific (within the course)
2. Change: intermediate and applied (in the learner’s work)
3. Impact: long-term and broad (organizational)

Kirkpatrick (1994) distinguishes between 4 types of evaluation:

1. Reaction
2. Learning
3. Behavior
4. Results

Berardinelli (1991) sees 3 elements to successful impact:

1. The quality of the learning experience (optimum design for learning, appropriate teacher activities)
2. The characteristics of the individual learner (begin with necessary levels of skill and ability, motivated and expects to be effective, can commit adequate time to learning, attempts to apply what has been learned)
3. The environment in which they apply what is learned afterwards (relationship between learner and supervisor, perceptions of rewards for enhanced performance)

Chapter 3:  “The Accountability Process and Planner”

(Note: This Accountability Planner takes the “Program Planning Process” and collapses the 6 stages into 3, and then adds 3 new stages)

1. Purpose and goals. What (content) and What for (objectives), including SKAs. Determine by needs assessment, organizational mandate, expert opinion, learner input.
2. Educational design. When, where, how. education process elements, including curriculum, learning tasks, materials, principles and practices, and instructors.
3. Anticipated changes. (learning, change, impact)
4. Evidence of change. Signs that change has occurred for each of the anticipated changes. Qualitative or quantitative, but direct, identifiable, specific and accessible. Likely to contain content and process elements.
5. Documentation of evidence. What sources to use? When should evidence be collected? Develop separate data collection instrument?
6. Analysis of evidence. What changes/gains did the program cause? Organize and summarize evidence. Analyze. Compare pre-course measures with post-course measures., comparison of participant and non-participant performance, comparison to an established standard, etc.

Chapter 4:  “Evaluating Existing Programs”

Difficult to evaluate programs already underway…extra time, effort, resources needed if not integrated from the beginning. Add-on evaluations may not be comprehensive or even appropriate for a given use. Externally-imposed evaluations may be resisted by the trainers…viewed as a threat.

Suggestions on how to “back in to evaluation”…

1. Prioritize evaluation needs…what information will be most helpful to improve the quality of the program? What is the primary purpose of the program?
2. Include major stakeholders in designing evaluation plan. Build “ownership” re: the value of evaluation.

Formal testing of learners is a narrow type of evaluation. Think “outside the box” regarding types of evidence and documentation of that evidence:

Evaluating Knowledge: Do they get it?

Oral questioning, discussions, writing assignments, class activities and assignments, work-related activities/assignments, case studies, flip chart, problem-solving.

Evaluating Skills: Can they do it?

Simulations of individual skills or complete tasks, games, projects, role plays. Actual performance in the work place. Observations of skills completed by the instructor, learning peers, supervisors, etc. using checklists of tasks or performance elements, rating scales, comparative rankings.

Evaluating Attitudes: Do they own it?

Self-perceptions. Perceptions of others (customers, coworkers, peers, instructors). Completed by direct observation, discreet observation or recall of past experience using listings/descriptions of desired attitudes and behaviors.

Much evaluation info is included in materials completed during the program. These can be collected and then reviewed later.

Additional suggestions on non-traditional ways to do assessment.

(From The Dirt on Learning, by Thom and Joni Schultz, Group Publishing, Loveland, CO., for church youth workers)

WAYS TO ASSESS FOR UNDERSTANDING, RETENTION, AND APPLICATION

1. observation
2. verbal responses
3. written records
4. drawing
5. products
6. self-evaluation tools
7. portfolios
8. teacher-student conferences
9. parent-teacher-student conferences
10. small-group conferences
11. journals
12. class scrapbook
13. faith history project
14. video projects
15. audio projects
16. living Bible museum
17. story box
18. dramatic presentation
19. living Bible verses
20. individualized educational programs
21. teacher for a day
22. music
23. show and tell
24. role play
25. “why” circle

Introduction

Every year a dizzying array of non-formal leadership training programs are offered worldwide. Some of these efforts use Western trainers with local interpreters, others utilize national trainers, while others rely on residential missionaries. Some offerings focus on standardized, programmatic instruction whereas others are less structured. Some efforts seek to replicate a full Bible institute curriculum while others are more targeted, focusing on a particular area of ministry and/or personal development. Some are one-time conferences or seminars while others are on-going, longitudinal efforts. Some depend heavily on technological resources while others rely primarily on either human or printed resources. Some invest heavily in training trainers, others less so. Some have strong institutional connections (e.g. Bible school extension programs), others are denomination-based, others local church-based, and yet others mission or agency-based.

What are we to make of this plethora of training approaches and efforts? On one hand, we rejoice that the Lord has raised up so many to meet the urgent need for church leadership development. Martin Luther, the great reformer, saw equipped pastors as the key to Church health and growth. “I entertain no sorry picture of our Church, but rather that of the Church flourishing through pure and uncorrupted teaching and one increasing with excellent ministers from day to day.”

On the other hand, one wonders, given all of this enormous effort: what are we accomplishing? The foundations and donors who support these ministries are wondering the same. Though a few training programs have clearly defined goals and measurable outcomes, many do not. Assessment and evaluation of non-formal leadership training tends to be sporadic at best.

At the first TOPIC (Trainers of Pastors International Coalition) consultation (Wheaton, December 1997) Dr. Mark Young questioned the efficacy of seminar/conference ministries in Eastern Europe. He suggested that perhaps our non-formal seminars are not as effective as we hope. Dr. Young then asked the provocative question, “Are we confusing response with results?” Response – those immediate and typically overwhelmingly favorable reactions to our ministries may be far different from results – the long-lasting fruit.

Our ministry, Leadership Resources International, equips and encourages pastors around the world to teach God’s Word with God’s heart. We seek to discover what’s really happening as a result of our ministry – be it good, bad, or ugly – and have built research and evaluation into the fabric of what we do. Our commitment to self-assessment has both a personal as well as a theological grounding.

The Question Guy

My earliest memory etched itself onto my 3-year-old brain as I was sitting on a stoop outside our 12-flat apartment building on the south side of Chicago. I remember a young toddler swaying to a halt in front of me. I looked up and simply asked: “Will you be my friend?” My very first memory centered on a question – a pretty good question at that.

Later – perhaps when in 1st grade – I remember asking my second question. Walking home from school with Helen, I stopped and asked her the big question. “Helen, when we grow up, will you marry me?” I’m not sure what came over me…perhaps I thought that you had to get your order in early or they’d run out of girls. As we all know, six-year-old girls are much more mature than six-year-old boys. Helen wisely suggested that perhaps we should wait a while to see what happens.

Have you noticed how important questions are in life? I wonder how many questions you have asked today: When’s breakfast? Who’s speaking? Is so and so here? When’s lunch? Where did I put my notes? Why does he need that? What’s for supper? Some of our questions are of little significance, while others carry enormous weight: What did the doctor say? Will she ever forgive me? How long does he have to live? Questions are an essential part of life. We could well say, “To live is to ask.”

I’m a professional question guy, having spent 15 years working for a survey research company. Our clients hired us to ask questions. Why would companies pay us huge amounts of money to simply ask questions?

My largest client was the Ford Motor Company. Car companies begin designing new cars and trucks four or five years in advance of actual production. They would build full-size clay and plastic models as part of the product development cycle and then have us invite two or three hundred people into a showroom to evaluate these futuristic models. We would ask tough questions. Ford wanted to know what people really thought about their advanced products, because they wanted to know how to improve the design. They wanted designs that would maximize sales.

Some of the auto designers were not very happy if they received criticism and negative feedback. On the other hand, the Ford managers wanted the truth—favorable or unfavorable—for the stakes were too high for anything less than a full and honest appraisal. After all, they were stewards of the company’s resources. The board of directors was expecting the managers of the company to make a profit, so Ford was willing to spend a few hundred thousand research dollars in order to make a good 300 million dollar product decision. They did not consider the research expenditure an expense, but rather an investment.

The Lord of Questions

If questions are important for corporations, they are crucial for the Kingdom. Have you noticed how often Jesus asked questions? Our Lord asked 99 questions in the gospel of Matthew alone. Why so many questions? The Scriptures suggest at least two major reasons. Firstly, Jesus, as a master teacher, knew that questions are a compelling teaching method. Questions force people to grapple with issues, moving them from passive listeners to active learners. Secondly, by asking questions, Jesus learned what the Father was doing.

Jesus’ ministry strategy was to watch for what the Father was doing. He only said and did what he heard and saw the Father doing. “The Son can do nothing by himself; he can only do what he sees the Father doing.” (John 5:19) “I do nothing on my own but speak just what the Father has taught me.” (John 8:28) This was Jesus’ “philosophy of ministry.” Jesus lived and ministered by watching to see what his Father was doing. He kept looking to see where and how the Father was leading. The Father was the initiator, the Son was the responder.

Think about the remarkable words that Jesus spoke on the cross: “It is finished” (John 19:30). How could he say that? For every blind man he healed, there were ten more still groping in the darkness. For every disciple that followed him, there were thousands more who did not. How could he declare, “It is finished?” The answer to this question is found in Jesus’ high-priestly prayer, “I have brought you glory on earth by completing the work you gave me to do.” (John 17:4) Jesus completed everything the Father had given Him to do.

How did Jesus learn what the Father was doing and what the Father was calling him to do? Of course, it began with prayer. Jesus spent extended time praying, especially when faced with major decisions and before and after momentous events. Jesus listened to the Father by reading and studying and memorizing the Scriptures. Jesus was also a keen observer – he spent time watching the people and the opportunities that the Father providentially brought into his life. Jesus’ understanding of what the Father was doing was shaped as He prayed, meditated on the Scriptures, and observed people and situations. But there was one other way that Jesus discerned what the Father was doing: he asked tough questions.

The Turning Point

Matthew 16 represents a pivotal time in Jesus’ ministry. Jesus poured his life into the disciples for three years, to seemingly no avail. He was perplexed by the density of the disciples’ hearts and minds. We hear Jesus’ exasperation expressed in Matthew 15:16, “Are you still so dull?” Listen to his words in Matthew 16:8-10 as he fires one question after another at his bewildered disciples: “You of little faith, why are you talking about…bread? Do you still not understand? Don’t you remember…? How is it that you don’t understand…?” They still don’t get it. Perhaps he’s wondering if they will ever get it.

Jesus is watching to see what the Father is doing. It must seem as though the Father is not working in the lives of the disciples. Yet, Jesus knows better – it’s not a matter of “if” but “when?” When will the three years of discipling his followers begin to bear fruit? When will his teaching and healing ministry be completed and his ministry of suffering and death begin? Jesus depended upon the Father’s timing.

In Matthew 16:13, Jesus travels with his disciples outside of Israel, to the region of Caesarea Philippi, northeast of Galilee. As far as we know, Jesus had not been in this area before. Jesus begins a little research project by asking the disciples, “Who do the people say that I am?” (v.13) He may be asking about the people in this new “market” or perhaps he’s referring to those back in Galilee. Either way, he wants to know how the people view him. “Some say John the Baptist; others say Elijah; and still others, Jeremiah or one of the prophets.” (v.14) These comparisons, though probably meant as words of encouragement, are far off the mark. After all this time, the people still don’t know who Jesus is. Have you ever taught or preached over an extended period of time and then discovered that your audience just wasn’t “getting it?” We don’t know how Jesus responded, but we can imagine him slowly and sadly shaking his head. Yet, Jesus has learning something from his question: the Father has not yet revealed to the crowds the reality of who Jesus is.

Jesus follows up this first question with two others that penetrate to the core: “But what about you? Who do you say I am?” Simon Peter responds with the famous affirmation, “You are the Christ, the Son of the living God.” (vs.15-16)

We can almost hear Jesus shout “Hallelujah!” Jesus discovered that the Father is working in his disciples after all. After Peter responds with “the right answer,” Jesus replies, “Blessed are you, Simon, son of Jonah, for this was not revealed to you by man, but by my Father in heaven.” (v.17) By asking a series of questions, Jesus sees more clearly what the Father has been doing – creating understanding and faith in the minds and hearts of the disciples after all. They finally are “getting it!”

The implications of this discovery are enormous, for it marks the beginning of the end. Jesus was always looking for the Father’s time – the “kairos” time. John tells us, “From that time on Jesus began to explain to his disciples that he must go to Jerusalem and suffer many things at the hands of the elders, chief priests and teachers of the law, and that he must be killed and on the third day be raised to life.” (v.21) Luke says simply, “As the time approached for him to be taken up to heaven, Jesus resolutely set out for Jerusalem.” (Luke 9:51) The disciples, the future Church leaders, finally understand. It’s time for Jesus to head to Jerusalem and the cross.

Here’s the point: through a series of thoughtful questions, Jesus discovered what the Father was doing and what he, as the Father’s Son, should be doing in response to his Father’s activity.

The BIG Question

We see the early Church asking the same question. The church in Jerusalem heard rumors about what was happening in Antioch and wondered, “What’s God up to?” So they sent up a researcher by the name of Barnabas. Another researcher, Luke, records that “When he arrived and saw the evidence of the grace of God, he was glad and encouraged them all to remain true to the Lord with all their hearts.” (Acts: 11:23) I love that phrase – evidences of the grace of God. I wonder what Barnabas actually saw. What were these “evidences” of God at work that Barnabas was able to see? There was something tangible, observable about God at work at Antioch. The church at Antioch was not only bearing fruit that would remain, but also fruit that could be seen.

Barnabas then asked himself the important follow-up question: Given what the Father is doing, what should I be doing? Barnabas saw exciting church growth at Antioch, but he also saw needs, especially the need for solid teaching of the Scriptures to establish the disciples and develop leaders.

We know that Barnabas saw large numbers of new believers (Luke mentions this twice.) Church growth is surely an “evidence of the grace of God.” I’ve often thought that church planters have it easier than pastor trainers and leader developers on this dimension – their “success” is more easily measured. Conversions, baptisms, church membership, and new church plants are all quantifiable. On the other hand, measuring spiritual growth and the development of leadership qualities and preaching and teaching skills is more complicated (but no less important.)

Barnabas saw what the Father was doing, but also what the Father had not yet done – what was left undone. So how did Barnabas respond? He went to Tarsus and found Saul and “so for a whole year Barnabas and Saul met with the church and taught great numbers of people.” (Acts 11:26) Like Jesus, Barnabas watched to see what the Father was doing and then walked in these things. In response to what they saw (and what they didn’t see!) in the Antioch church, Barnabas and Saul equipped and encouraged a team of pastoral leaders from Africa, Asia and Europe (Acts 13:1) which then set the stage for the first sustained missionary effort of the New Testament Church.

Questions for Discussion and Reflection

• What is God doing in and through your ministry?
• What fruit is he producing through you?
• What are the evidences of the grace of God in your ministry?
• How do you evaluate your own teaching and training efforts?
• How do you discern what the Lord is doing?
• What questions should you be asking?

Seeing but the Outer Fringes

Why is it so difficult to discern what God is doing in and through our ministries? The combination of God’s vastness and our smallness prevent us from clearly seeing what God is doing. If indeed God’s activity is infinite and Jesus is “sustaining all things by his powerful word” (Heb 1:3) and the Father “is always at work” (John 5:17), then we should not be surprised by our inability to grasp the works and wonders of God.

There’s so much we don’t know. The Father has chosen to reveal to us only a small portion of what He is doing. Job describes the unfathomable workings of God in the heavens and upon the earth, in both the skies as well as the seas, and then concludes, “And these are but the outer fringe of his works; how faint the whisper we hear of him! Who then can understand the thunder of his power?” (Job 26:14)

Think of all our human limitations which hamper our ability to discern his ways. We are locked into the here and the now – the past is a fog and the future is unknown. We are embodied creatures, rooted in and oriented toward the physical dimensions of life. Our “bentness” toward sin and self-centeredness form additional barriers that impede us from clearly seeing all that God is doing. We simply don’t have the capacity to fully comprehend his works.

Consider this analogy from science. All energy is expressed as electromagnetic radiation in the form of waves—ultraviolet, x-rays, light waves, and many others. At one end of the wave spectrum are cosmic radio waves – each one is more than a kilometer long. At the other end of the spectrum are tiny gamma rays – each one less than one micron (one millionth of a meter.) In between are others waves with varying frequencies.

Humans only have the capacity to experience a small portion of the wave spectrum: visible light. Though constantly bombarded by waves of waves, we don’t realize it because we don’t have the capacity (i.e. “the tools”) to monitor or observe these. God’s activity on earth is similar: his work is constant. In our days, he is holding all of the atoms of the universe together, providentially working his purposes in the lives of individuals and families, raising up and tearing down leaders and nations, and sovereignly directing history toward his good purposes. Yet, we only catch glimpses of what he is doing. Every so often he graciously lifts the veil to show us some of what he is doing.

“How is God using us?” is a perplexing question that’s tough to answer. Tough, but not impossible. Barnabas saw the evidences of God at work. He didn’t see everything God was doing, but he saw something – enough to recognize God’s handiwork. Barnabas caught glimpses of what God was doing.

Following are some practical suggestions to help you discern some of what the Father is doing in and through your ministry. We’ve also developed a simple tool (Leadership Resources’ Program Design Grid) to help establish clear goals and appropriate ways to evaluate what’s really happening. Though designed for pastor-trainers, it can easily be adapted for other cross-cultural ministries.

Three Other Questions

Three questions can help us answer the BIG question: what is God doing? This set of questions can be asked about a course you teach, a group of pastors that you are equipping, or any other ministry effort. As you think about a specific area of your ministry, ask:

1. What do I hope to see happen?

A question of intentionality. Too often we undertake ministry without a clear sense of the specific results that are we aiming for. Have we defined our goals and intended outcomes? For example, think about an upcoming pastor training course. You’re offering this course for a reason – you want something to happen. There’s a response, a change that you’re looking for – a “so what?” As you prepare, you might ask: How specifically do I want pastors to respond to these lessons? What impact do I want to see in their walk with God, in their relationships, in their heart for ministry, in their ministry competencies, etc? What long-term changes do I hope to see?

Here’s another way of thinking about it: if the Lord were to answer all of your prayers for this training course, what would happen? What would it look like in the lives of the individuals you are equipping, and their churches, and their communities…next month, next year, ten years from now? What glorious transformations do you believe that the Lord wants to work through this specific training effort?

2. Are these things actually happening?

A question of impact. Too often we assume transformation in those leaders we are equipping, without really knowing. Did the things you were praying for and working towards happen? How do you know – what are the indicators, the “evidences of the grace of God,” that you see? What did you see God do – and not do – at least as far as you can tell? This question begs for both humility and honesty.

3. What should I do differently in the future?

A question of learning—not so much what others learned—but what we, as the teachers or trainers or church planters or ministry providers, learned. Of course, God desires to transform others through us, but he’s just as committed to transforming you and me in the process. Every ministry opportunity is a chance for us to grow, to get sharper, and to become more effective. After every pastoral training event our staff undertakes, I encourage our team to first rejoice in what the Lord accomplished, and then ask, “What did I learn? What worked? What didn’t work so well? What needs to change so that I can be even more effective next time?” These are the questions of a steward seeking to serve his master well. They, too, call for honest humility.

Keep in mind that these three subordinate questions serve the first question: what is God doing? We need to set goals, but goals that flow out of what we see the Father doing. God’s goals are far more precious than our own. Ecclesiastes 3:14 reminds us, “I know that everything God does will endure forever; nothing can be added to it and nothing taken away from it. God does it so men will revere him.” If our ministry flows out of our dreams, the effects won’t last. Only what the Father does lasts forever. We do not initiate ministry – we walk in the things we see the Father doing. Like Jesus and like Barnabas, we are to do what we see the Father doing, we are to speak the words we hear our Father speaking, we are to align ourselves with His providential purposes.

In Summary

Asking tough questions is essential to effective ministry. The most important question is this: What is the Father doing? But the follow-on questions are critical as well. Learn to ask tough questions. What kinds of changes are really happening in the lives of those we teach and serve? What kinds of changes should we consider in terms of content, technique, focus, skills, and so on? Surely our ministry efforts will have greater impact, as we become…

• More reflective – asking ourselves the hard, poignant questions
• More intentional – what are we really trying to accomplish?
• More informed – how do we know what we are accomplishing?
• More flexible – how ought we change in order to improve our ministry?

Remember the electromagnetic radiation spectrum? We have physical limitations that allow us to only see and experience a small portion of it? However, scientists have developed special tools to monitor other portions of the spectrum. Amplifiers, special scopes, satellite dishes and other types of technology enhance our capacity to “see” more of the wave spectrum. Asking tough questions is a way for us, as servants of God, to see more of the spectrum of what God is doing in and through us.

Tough, thoughtful, penetrating questions can become tools to help us observe “the evidences of the grace of God.” Might the Lord give each of us the wisdom to ask the right questions, the honesty to search carefully for the answers, and then the humility and courage to change, so that the Church of our Lord Jesus Christ might flourish more fully, to the glory of God.

Craig Parro
President
Leadership Resources International

Dr. Mark Young asked a group of pastor-trainers the provocative question, “are we confusing responses with results?” after telling this poignant (and unfortunately, true) story.

A prominent American pastor traveled with a small entourage to Poland to teach a pastors’ conference. Some time afterwards, Mark asked one of his Polish friends who had attended the conference about it. “It was great!” was his enthusiastic response. “Yes,” he continued, “all of us pastors loved it – we got to spend a weekend in a 4-star hotel. And all the Americans loved it – they got to spend a lot of money.”

In my mind’s eye, I envision the American pastor reporting back to his congregation the warm response he received in Poland…the words of gratitude, the tears in the eyes, the long and strong bear-hugs, and the invitations to come back soon.

“Are we confusing responses with results?”

Background

Leadership Resources International and MasterWorks Foundation hosted a 24-hour, invitation-only summit in November 2009 in Chicago on the subject, “Best Practices” for Non-formal Pastoral Training Agencies. The overall goal was to help practitioners, through dialogue with informed and experienced peers, identify ways (best practices) to more effectively evaluate their training ministries as they seek to equip shepherds for the Church of our Lord Jesus Christ.

The 16 participants were (and are) leaders in the field—either principals from non-formal pastoral training agencies or representatives from foundations that have a particular passion for leadership development and pastoral training. The training agencies represented included BILD International, Entrust, Russian Ministries, Africa Ministries Network (AFMIN), Leader Source, Development Associates International (DAI) and Leadership Resources International (LRI). The foundations represented were: James C. Blankenmeyer Foundation, Cornerstone Trust, First Fruits Inc., Harry J. Lloyd Trust, Maclellan Family Foundations, and MasterWorks Foundation.

In addition, TOPIC (Trainers of Pastors International Coalition) was represented. TOPIC, as a global coalition of several hundred non-formal pastoral trainers and their organizations, has a keen interest in helping establish and promote “industry” best practices [www.TOPIC.US]. Overseas Council, an organization that works with Bible institutes and seminaries worldwide, also participated.

Focus of the Conversation

Through a collaborative process, the summit participants agreed to focus the conversation on pastoral training evaluation—identifying the best assessment and improvement practices. (The focus of this roundtable was on developing pastoral leaders rather than the broader topic of leadership development.)

Questions and issues raised on the subject of evaluation and assessment included…

• What is a biblical basis for evaluation? How did Jesus evaluate?
• Integrating evaluation into the training methodology (vs. as an after-thought).
• Inculcating a continuous improvement mindset—“How are we doing? And, how can we get better?”
• Maintaining the quality of training, especially to the 2nd/3rd generation.
• Appropriate ways to measure different types of outcomes, such as…
  • Knowledge/theology/wisdom/worldview
  • Skills/competencies
  • Character/integrity/humility/“on-going brokenness”
  • Love/heart/passion/will
  • Spiritual formation/habits
• What can we rightfully “take credit” for? How certain are we that our training is (solely) responsible for the changes that we observe? (The problem of multiple explanatory variables.)
• Obtaining “real” numbers (vs. “evangelistically-speaking”).
• Appropriate use of anecdotes.
• Cost/benefit—are donations being “leveraged?”
• How do donor goals mesh with agency goals? What role should donors play in establishing outcomes and measurement systems?

The day of dialogue was marked by attentive listening with lots of give and take, vulnerability and collegiality. Often, there were more questions than answers. Nevertheless, we were able to identify 15 “best practices” for pastor trainers and pastoral training organizations as we evaluate our training efforts.

Summary of the Findings

1. Evaluation as part of the organizational DNA

All agreed that evaluation and assessment were critical values that training organizations need to embrace.

“Best practice” pastoral training organizations will…

• Create and maintain a “culture of reflection,” a community of self-reflecting leaders.
• Develop a “holy discontent” that results in a commitment to continuous improvement.
• Self-assess their ministry from start to finish, including strategy, design, process, and implementation.
• Weave evaluation throughout the teaching and training times, and not simply as a brief “add on” at the end.

2. Establishing clear goals and outcomes

All agreed that effective evaluation is dependent on establishing clear goals. However, differences emerged regarding the appropriate scope of the goals. Namely, should the goals we pursue be…

Comprehensive or narrow?

The changes that training organizations seek to stimulate might be visualized as concentric circles: the more immediate changes we seek (the inner circles) often relate to the learner’s knowledge, competency, and Christ-likeness. Broader, more holistic spheres of change might include expanded organizational capacity, multiplying churches and transformed communities. On one level, of course, we want our training to ultimately result in these larger changes. On the other hand, the broader the changes, the greater the difficulty we face directly connecting our training to those changes. Big, overarching outcomes often have multiple explanatory variables. By way of example: to what extent was the significant church growth that we observed the direct result of our training? And/or, to what extent was it the result of many other influences in church and society (e.g. a migration influx, a widely-known miraculous healing, the arrival of an anointed evangelist, improved formal training institution programs, etc.)?

Two different organizations participating in the roundtable shared different approaches that they have taken on this question of comprehensive vs. narrow scope. LeaderSource has developed a comprehensive survey-based leader evaluation system called 5C CheckPoint which assesses: Christ, Character, Community, Calling, and Competency. They have created 18 indicators for each “C” with questions behind each indicator. The model includes both self-evaluation as well as evaluation by others, enabling an in-depth, holistic evaluation of the leader. [www.5ccheckpoint.com]

Leadership Resources International (LRI) embraces a narrower approach to evaluation tied to a set of tightly defined goals. LRI training goals are to teach pastors how to study, teach and preach God’s Word with God’s heart, and then multiply that training to other pastors and leaders. A group of 10-15 pastors form a given training cohort which meets with an LRI team multiple times over a 4-year period. In this highly-relational setting, evaluation actually becomes a part of the teaching and mentoring process. The combination of practice, interaction, oral and written reports, intentional observations, statistical tracking, and self-evaluation enables LRI to encourage the pastors while, at the same time, track their progress over several years. LRI evaluation focuses on those competencies and heart issues which are the goals of the training as well as on the multiplication of the training (to the 2nd & 3rd generations) and resulting church growth.

Longer vs. shorter timeframes?

All pastoral training organizations want to achieve long-term results that far outlive the actual training sessions. But at what point do we make a definitive assessment of our training program–after 1 year, 5 years, 1 generation, or ??? The longer the timeframe, the greater is the confounding effect of multiple explanatory variables.

The twin issues of scope and timeframe merged in a discussion of outputs and outcomes. “Outputs” were defined as the quantifiable results that we seek. Outputs tend to be more immediate and more observable—knowledge gained and competencies demonstrated. But should we be satisfied with outputs? Don’t these outputs actually serve a higher cause, namely, “outcomes?” Outcomes are often more qualitative and anecdotal (e.g. heart changes, healed relationships, spiritual growth, character transformation, organizational impact, healthy multiplying churches, transformed communities), yet more significant in that they point to deeper and longer-lasting impacts. Outcomes are the profound, transformative changes which emerge over a period of years or even generations.

Evaluating broadly-defined and long-term outcomes can lead to a highly complex evaluation design. Yet, for an evaluation plan to be sustainable, especially at the field level where the information is collected, the data collection tools and procedures need to be kept simple. The challenges of geography, security, lack of technology, cultural and language barriers, and literacy challenges all argue for a “keep it simple” approach to evaluation. Clearly a tension exists between the twin desires for comprehensiveness and simplicity.

How then do we evaluate outcomes and the relationship of our training to those outcomes?

“Best practice” pastoral training organizations will…

• Clearly define their goals in terms of both outputs (i.e., near-term learning including skill development and utilization) and outcomes (long-term, broad transformations sought).
• Not be content to only evaluate inputs (i.e., what is taught,) but outputs (do they get it, can they do it?) and outcomes (how is the organization, church and community impacted?).
• Stay focused on evaluating long-term outcomes, recognizing that while long-term outcomes are more difficult to assess accurately, they are, in fact, the ultimate goal and justification for the training effort.
• Conduct longitudinal research where appropriate, obtaining multiple measurements along a timeline. When appropriate, begin with baseline/pre- measures.
• Keep the evaluation plan as simple as possible, taking into account data gathering challenges in the field.
• Evaluate with humility and honesty, keenly aware that God is using many other people, organizations, and situations to accomplish His work.

3. Maintaining relational integrity

Who determines what is to be learned? Who defines the teaching goals and corresponding changes sought—the teacher or the learner? Pastoral training organizations have a core set of competencies and a particular ministry calling. At the same time, learners have an understanding of their own training needs and desires.

“Best practice” pastoral training organizations will…

• dialogue with indigenous learners and leaders with “appreciative inquiry” to align the learners’ training needs with their own organizational training capacities and calling.

Who determines what should be evaluated? Who should do the evaluating? Evaluation may be viewed as a threat by learners, especially in shame-based cultures or oral-oriented societies. In worst case scenarios, evaluation will be seen as a tool of power, control or manipulation. The design of an evaluation plan should carefully reflect relational and “power” dynamics among the major stakeholders. Effective evaluation can and should be undertaken in a highly-relational, non-threatening way. Some of the best evaluation flows out of deep relationships. As one participant noted, “paper flow decreases as relationships increase.”

Evaluation and accountability should cut both ways—both teachers and learners will have valid opinions about the effectiveness of the training. Evaluation is a way for learners to discover what they learned (“how do they know they know?”) and a way for teachers to become better teachers—to discover what about their teaching and methodology was effective and helpful, and what was less so.

“Best practice” pastoral training organizations will…

• Dialogue with indigenous learners and leaders about the value of evaluation in order to encourage a culture of reflection and build ownership for an appropriate evaluation plan.
• Design an evaluation plan that includes a learner self-assessment component and other appropriate tools such as “growth portfolios” which document on-going growth and demonstrated competencies.
• Invite indigenous learners and leaders to evaluate the pastor trainers and the relevancy and effectiveness of the training that they bring.
• Approach evaluation in a relationally healthy way, sensitive to both cross-cultural realities and relational implications.

Conclusions

Participants identified 15 evaluation and assessment best practices for pastoral trainers and pastoral training agencies. The group clearly recognized the need for more learning in this area and that non-western trainers and training organizations need to be invited into the conversation.

Other pastoral training arenas deserving a best practices forum might include:

• Best Program Design Practices (e.g. selection of trainees, scalability and replicability)
• Best Cross-cultural Partnership Practices (creating and maintaining ownership and alignment)
• Best Curriculum Practices
• Best Training Practices for Oral-learners
• Best Pedagogy Practices (training methodologies)
• Best Administrative Practices

A final area of exploration that generated much interest was the fostering of healthy synergies between the worlds of formal and non-formal training.

Craig Parro
President
Leadership Resources International

Our TNT training is unique in its focus on multiplication. We don’t define success by our training of the TNTers – we have tons of evidence that those training times are effective.

Success for us is what happens when our TNTers train the next generation of local pastors – when those 2nd generation pastors and leaders are able to faithfully and skillfully teach God’s Word with God’s heart.

Listen as pastor “Noah” (a second generation TNTer) describes how God has used TNT training to revitalize his church through improved preaching and teaching.

Let’s see what happens when Americans get out of the way.
Leadership Resources is training pastors in the slums of Nairobi, Kenya through our Training National Trainers (TNT) program.

No one can impact a country for Jesus more than the people of that country. Hear about how a Kenyan pastor was gripped by a vision for training pastors in his own city.

Leadership Resources’ “Training National Trainers” Program (TNT) seeks to equip disadvantaged pastors across the globe to teach God’s Word with God’s heart.

The pastor in this video labors with few resources in an Asian country hostile to Christ and His Church.  Listen to how God is using TNT-trained pastors like him to transform churches around the world, developing congregations with a heart for missions.

Designed to share God’s love by helping some of the every day needs of the children in the churches and communities of our Training National Trainers (TNT), and bring a bit of hope to their lives. Our SEEDS of hope bag will include a Gospel presentation, a few personal hygiene items, some learning supplies, and a toy.

For just $25, you’ll be able to provide one SEEDS of hope bag that will extend God’s love and care to a special boy or girl that is being influenced by our TNTers.

This is a great way to get a small group or sunday school class introduced to our TNT program. Download the flyer to share all the details.