It has now been about 6 weeks since I performed a self-assessment of my competencies as an evaluator. My formal learning on the subject is coming to a close, so I intend to end it the same way I started, with a self-assessment. I’m going to be using the same criteria to judge myself now as I did then. However, the nature of my learning experience means I will be referencing an evaluation project to provide context for my growth. Additionally, since the learning primarily covered the mechanics of the field, many of the points I make will focus on will be similar to my previous post. Like before, I’ll be splitting the post into 5 pillars of evaluation competency while measuring my abilities in each.
First off, how would I rate myself as an evaluator on a scale of 1 to 6? I’d say, after the learning, I’ve done and the work I’ve performed, I’m about a 4 on the scale of novice to expert. For the most part, the improvements I’ve made are on a largely technical level. However, it is no exaggeration to say that what surprised me was the volume of technical information involved in the process of evaluation.
Professional Practice
One of the big things that I didn’t consider during my last self-assessment was the Dunning-Kruger effect. I can say with certainty that the biggest step I’ve taken over the course of the last 6 weeks was the realization that I have a painful lack of experience and awareness of the many intricacies of evaluation. The variety and depth of systematic approaches and data available to me gave me a new appreciation for the people who make a profession out of the field. I also find myself again remarking on the similarity of the ethical side of things to lessons taught to me about adopting the perspectives of others to improve the work. Though, I now understand the terrifyingly large number of factors, both cultural and status based which can affect the course of an evaluation. Data can be obfuscated or falsified when people believe that any sort of “alteration” can lead to a negative outcome for them. Particularly, I find myself lacking experience in the subject of power dynamics. It’s difficult to say how well I might handle a situation where status plays a major role, but I can say that my aim would be to handle it in a manner that validates the system and the roles within it. In my experience, constructive ideas generate transformative effects rather than destructive ones. I don’t know how I would feel about an evaluation that recommends a program to be terminated.
Methodology
A significant improvement over my previous knowledge. The second edition of Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change is a great resource for understanding the different models, data sources, and procedures involved in the process of evaluation. A skill I honed, in particular, was the skill to determine the different types of information that would be useful to the evaluation and how to identify the important objectives and questions that were relevant to the evaluation. Luckily, I have a new copy of the book sitting in my Kindle library, rereading it is my new assignment since I think the information in the book may have some uses in Video Game QA.
Context Awareness
So I understand why the AEA gave so much focus to the “evaluation stakeholders” now. Here is a quick list of the things the stakeholders have a significant involvement or impact on within the evaluation process:
- The amount of information that we receive for our analysis of the program,
- The quality, and type of information that we receive,
- when we receive that information,
- the context and frequency of our progress report,
- and the format of the final presentation.
In addition to the factors I listed above, any recommended action on the part of the evaluators is entirely handled by the stakeholders. The most significant trick I learned in the last weeks is that involving the stakeholders in the evaluation process makes them more likely to accept the information they are presented with. Another personal assignment, take notes on how organizations “like” having their information presented.
Planning & Management
A simple part of the evaluation process that I underestimated was the importance of data and document management. When working with other people, it is incredibly important to sort all the information that is being used into one, easily accessible, and secure location; Google Drive was a blessing in this capacity. However, the spread of information between folders made the entire process of evaluation more stressful and tiring. Note to self, naming conventions are useful things to establish, regardless of what the project is addressing. Another thing I was happily surprised by was that my experience with game documentation meant that the formatting of the project report was both quick and aided in clarity.
Interpersonal Competence
I’m pretty humbled by this one. While my ability to work with others improved significantly over the course of the project. The significant weakness I demonstrated to myself is that I need to work on my oral presentation skills. I talk slow, molasses slow. In particular, I need to start practicing my ability to speak on my work to people who do not understand it. While I give myself plenty of time to think through what I say, I need to work on speaking with confidence and volume.
Evaluation is more complicated than I thought it would be.