I doubt I’m the only one who finds some measure of ironic humor in the practice of self-evaluating your capacities as an evaluator. My biases demand I rate myself as a novice on the scale of one to six that I’m looking at. But, given the nature of the field, I think I’ll start by putting those thoughts away so I can spend time talking about the many model competencies that one should expect from a good evaluator and how I measure up to them. Broadly speaking there are about five or six evaluator competencies to consider based on which article you are referencing. The 2018 AEA Evaluator Competencies says there are five, an article titled Establishing Essential Competencies for Program Evaluators, written by one Laurie Stevahn of Seattle University back in 2005 says six. For the most part, I’ll be referencing the 2018 competencies, but a few points from Laurie’s article will make their way in. Those competencies fall in the following domains:
- Professional Practice,
- Methodology,
- Context Awareness,
- Planning & Management,
- and Interpersonal Competence.
Professional Practice
Starting out, I can say that I hadn’t considered the professional or ethical implications of evaluation practices nearly enough. Nor did I consider the systematic qualities of the field. My personal homework is to begin researching various evaluation approaches and theories so I can begin to apply them in a more systematic and definable capacity. However, what surprised me most about the competencies of professional practice was the number of competencies that were easy to align with my way of thinking. For example, the practice of ethical evaluation was much easier to understand and find examples of when I considered how I try to evaluate people with the idea of “fairness” in mind. The majority of competencies in the domain of “Professional Practice” can be summarized as, “the consistent practice of evaluation, both personally and professionally, in a manner that is systematically driven by the principles of fairness.” Personally, I believe myself to be adept in this domain. Teachers and mentors instilled, long ago, the idea of personal accountability and the importance of reflection with goal-making in mind.
Methodology
This domain is the one in which my lack of professional experience glares damningly. My own experiences as a game designer have granted me some examples of this domain. However, a focus on qualitative evaluation has robbed me of insight into the more technical aspects that define the domain. The focus of my studies for this one domain is focused more on cultivating knowledge than acquiring perspective. Studying things like program logic and program theory would likely help greatly in improving my competency in this domain. However, beyond theory study, I think my greatest improvement would be in redesigning the evaluations. The most technical procedures have always escaped me in subjects like this so to improve my understanding of methodology it would likely be better to focus on practices that make the data easier to analyze. By designing the evaluations to reveal trends or with a heavy focus on context-sensitive data.
Context Awareness
An interesting domain, I was surprised by the amount of focus that the AEA gave to “evaluation stakeholders” in this domain. However, makes sense when related to the next two domains. Evaluation stakeholders include users and SMEs on top of providers. There is a great focus placed on the capacity to sustain multiple perspectives and effectively communicate with the various different parties that are involved in the generation of evaluations. One particularly notable competency in this domain focuses on the ability to describe the program, its components, and its functions in a broader context. My undergraduate program trained me to quickly and accurately adapt diverse perspectives on the subject matter. Facilitating a shared understanding of the program and its evaluation allows the stakeholders to accurately determine the effectiveness of the program as well as what the actual effect of the program is. I rate myself as adept in this domain
Planning & Management
Returning to game design for a minute, running QA evaluations is an arduous process. I’ll readily admit that I lack experience with coordinating and supervising the evaluation process of products. It’s important to understand the resources you have at your disposal so you can determine what is realistically feasible and move closer toward what is ideal. Unlike the other domains, I’m not sure how to improve the competencies in this one. I do, however, have talent when it comes to documentation. Particularly, I have a focus on effective documentation of the evaluation processes and plans so they can be understood rapidly by others. My work for improvement outside of the course would be more focused on competencies that focus on determining when and how to work with others while also taking into consideration how the evaluation influences organizational action.
Interpersonal Competence
Hardly a surprising domain, given the focus on stakeholders and mindfulness of both cultural and personal impact. A competency I have particular confidence in is the ability to listen to, understand, and engage in different perspectives. One surprise, however, was the idea of building trust throughout the evaluation. As a general rule, I found the practice of evaluation itself to be rather unpleasant. I’m interested in the idea of using it as a tool to build rapport. However, for my own improvement, I’d like to focus on the last competency, constructive conflict management. Something to focus on later.
For now, let’s wrap up by going back to that scale from before. 1-6 scale of evaluation competency. I’d go for ~3. Maybe I’m selling myself short I’ll come back to this After a bit more learning and see where I am then.