9 November 2017
I frequently ask people for all types of feedback:
- I request in-depth thoughts and data from playtesters of our games.
- I talk to a variety of people–including readers of this blog–about ways Stonemaier can improve as a company.
- Whenever I have a new project page to launch, I seek input from ambassadors to ensure the page is clear and appealing.
There was a time when I thought “more is better” when it comes to feedback. But I’ve realized that giving feedback is a skill. Some people are good at it, and others aren’t. In most cases, I’d rather have the input of 10 people who are skilled in the art of feedback than 100 people who don’t have those skills.
So a few months ago, I decided to put together a little quiz to see which of our Stonemaier ambassadors were ideal playtesters. I then shared the results with all ambassadors (they were curious about the “correct” answers) and also gave them a second chance to participate in future playtests if they understood what I was looking for. Some of these skills are innate, while others can be learned and honed.
Even though this quiz is focused on tabletop game playtesting, I think the core concepts here could apply to any product testing, company feedback, or Kickstarter project page input. My answers are noted in italics below each question.
1. If you playtest a game and find a mechanism you really don’t enjoy, which of the following phrases are you most likely to say to the designer? I’ll use the dice-rolling mechanism in Catan as an example.
a. I see what you are trying to do with the dice-rolling mechanism, and I like rolling dice, but it needs some work.
b. The dice-rolling mechanism is terrible.
c. One playtester didn’t have settlements on any 6 or 8 territories, yet he was able to produce resources nearly every turn because the 6 and 8 were hardly ever rolled. He was happy, but the other playtesters were frustrated that the high-odds numbers they had wagered on reaped very few benefits.
d. The dice-rolling mechanism is way too random. I would suggest replacing it with a deck of cards.
My correct answer is (c). The question was designed to test how you express feedback. By far the most useful playtest feedback we receive is feedback that describes WHAT happened and WHY it was frustrating/confusing/boring, etc. That’s the information I need to understand the problem and find a solution.
2. You’re in the middle of a playtest and the deck of cards is empty. The rulebook doesn’t mention this situation. What do you do?
a. shuffle the discard pile to create a new deck and draw from it
b. e-mail Jamey and wait to proceed until you have an answer
c. do not draw cards from that deck for the rest of the game
My correct answer is (a). The question was designed to test your intuition. When you playtest a game, you’re working with an in-progress rulebook in real time, so you need to be able to intuit the intended meaning without getting hung up. You can note the confusion in your feedback later. In this example, the only reason you wouldn’t reshuffle the deck is if the rules go out of their way to tell you not to reshuffle.
3. One of your playtesters has used the exact same strategy for two games in a row with nearly identical results. What would you do?
a. nothing – part of playtesting is allowing players to freely explore strategies as they wish
b. after the second game, mention to the playtester in private that you’re glad they’ve deeply probed a strategy but that they need to try something different next time
My correct answer is (b). The question was designed to test your leadership. You’re going to see playtesters do things that are really helpful and insightful (like breaking the game)…and you’re going to see playtesters do stuff that significantly detracts from the value of the playtest (like breaking the game the same way every time they play). It’s your responsibility as the lead playtester to step in and get the most value out of each playtest. Sometimes that may even mean that you tell one player to pursue Strategy A and another player to pursue Strategy B.
4. The playtest period is ending soon and you’ve only played 1 out of 4 games. You don’t think you can play the remaining 3 games in the allotted time. What do you do?
a. email Jamey and ask for a 1-week extension
b. end the playtest early
c. rearrange your schedule and complete the remaining 3 games in a marathon session
d. this question doesn’t apply to you because you would never wait until the last minute to complete the majority of something you committed to do
My correct answer is (d). The question was designed to test your commitment. If you commit to a playtest, we expect you to follow through. While I appreciate people who answered (c), and I’m fine with people playing back-to-back games, I’m looking for people who can plan ahead and mitigate the unknown future instead of those who cram at the last minute.
5. Which of the following would you be the LEAST excited to do if given the option?
a. film and upload unedited videos of several playtest sessions
b. write a detailed, lengthy description of the playtest experience
c. record an audio or video discussion with playtesters after the playtests are complete
My correct answers are (a) and (c). The question was designed to test your willingness to write long-form feedback. This question is a little odd because it’s just about personal preference. I’ve tried receiving feedback in video and audio form, but I best process feedback if it’s written down, so I need to find people who are willing and comfortable with long-form writing.
6. After completing a playtest, the feedback form asks you to describe the most frustrating or confusing thing that happened during the game. Could you see yourself ever leaving that answer blank after a blind playtest?
a. yes, I would leave it blank if the playtest went really well
b. no, I could always find something frustrating or confusing to share
c. if the playtest went really well, I would explain why I thought it worked in lieu of listing things that frustrated or confused me
My correct answers are (b) and (c). This question was designed to test your awareness. The only feedback that’s useless to me is no feedback at all. There’s always something to say, and I’m interested in lead playtesters who are constantly processing each playtest session and are able to put their observations into words.
What questions would you ask to help you select the best testers?