Kickstarter Lesson #236: Selecting the Best Testers

9 November 2017

I frequently ask people for all types of feedback:

  • I request in-depth thoughts and data from playtesters of our games.
  • I talk to a variety of people–including readers of this blog–about ways Stonemaier can improve as a company.
  • Whenever I have a new project page to launch, I seek input from ambassadors to ensure the page is clear and appealing.

There was a time when I thought “more is better” when it comes to feedback. But I’ve realized that giving feedback is a skill. Some people are good at it, and others aren’t. In most cases, I’d rather have the input of 10 people who are skilled in the art of feedback than 100 people who don’t have those skills.

So a few months ago, I decided to put together a little quiz to see which of our Stonemaier ambassadors were ideal playtesters. I then shared the results with all ambassadors (they were curious about the “correct” answers) and also gave them a second chance to participate in future playtests if they understood what I was looking for. Some of these skills are innate, while others can be learned and honed.

Even though this quiz is focused on tabletop game playtesting, I think the core concepts here could apply to any product testing, company feedback, or Kickstarter project page input. My answers are noted in italics below each question.

1. If you playtest a game and find a mechanism you really don’t enjoy, which of the following phrases are you most likely to say to the designer? I’ll use the dice-rolling mechanism in Catan as an example.

a. I see what you are trying to do with the dice-rolling mechanism, and I like rolling dice, but it needs some work.
b. The dice-rolling mechanism is terrible.
c. One playtester didn’t have settlements on any 6 or 8 territories, yet he was able to produce resources nearly every turn because the 6 and 8 were hardly ever rolled. He was happy, but the other playtesters were frustrated that the high-odds numbers they had wagered on reaped very few benefits.
d. The dice-rolling mechanism is way too random. I would suggest replacing it with a deck of cards.

My correct answer is (c). The question was designed to test how you express feedback. By far the most useful playtest feedback we receive is feedback that describes WHAT happened and WHY it was frustrating/confusing/boring, etc. That’s the information I need to understand the problem and find a solution.

2. You’re in the middle of a playtest and the deck of cards is empty. The rulebook doesn’t mention this situation. What do you do?

a. shuffle the discard pile to create a new deck and draw from it
b. e-mail Jamey and wait to proceed until you have an answer
c. do not draw cards from that deck for the rest of the game

My correct answer is (a). The question was designed to test your intuition. When you playtest a game, you’re working with an in-progress rulebook in real time, so you need to be able to intuit the intended meaning without getting hung up. You can note the confusion in your feedback later. In this example, the only reason you wouldn’t reshuffle the deck is if the rules go out of their way to tell you not to reshuffle.

3. One of your playtesters has used the exact same strategy for two games in a row with nearly identical results. What would you do?

a. nothing – part of playtesting is allowing players to freely explore strategies as they wish
b. after the second game, mention to the playtester in private that you’re glad they’ve deeply probed a strategy but that they need to try something different next time

My correct answer is (b). The question was designed to test your leadership. You’re going to see playtesters do things that are really helpful and insightful (like breaking the game)…and you’re going to see playtesters do stuff that significantly detracts from the value of the playtest (like breaking the game the same way every time they play). It’s your responsibility as the lead playtester to step in and get the most value out of each playtest. Sometimes that may even mean that you tell one player to pursue Strategy A and another player to pursue Strategy B.

4. The playtest period is ending soon and you’ve only played 1 out of 4 games. You don’t think you can play the remaining 3 games in the allotted time. What do you do?

a. email Jamey and ask for a 1-week extension
b. end the playtest early
c. rearrange your schedule and complete the remaining 3 games in a marathon session
d. this question doesn’t apply to you because you would never wait until the last minute to complete the majority of something you committed to do

My correct answer is (d). The question was designed to test your commitment. If you commit to a playtest, we expect you to follow through. While I appreciate people who answered (c), and I’m fine with people playing back-to-back games, I’m looking for people who can plan ahead and mitigate the unknown future instead of those who cram at the last minute.

5. Which of the following would you be the LEAST excited to do if given the option?

a. film and upload unedited videos of several playtest sessions
b. write a detailed, lengthy description of the playtest experience
c. record an audio or video discussion with playtesters after the playtests are complete

My correct answers are (a) and (c). The question was designed to test your willingness to write long-form feedback. This question is a little odd because it’s just about personal preference. I’ve tried receiving feedback in video and audio form, but I best process feedback if it’s written down, so I need to find people who are willing and comfortable with long-form writing.

6. After completing a playtest, the feedback form asks you to describe the most frustrating or confusing thing that happened during the game. Could you see yourself ever leaving that answer blank after a blind playtest?

a. yes, I would leave it blank if the playtest went really well
b. no, I could always find something frustrating or confusing to share
c. if the playtest went really well, I would explain why I thought it worked in lieu of listing things that frustrated or confused me

My correct answers are (b) and (c). This question was designed to test your awareness. The only feedback that’s useless to me is no feedback at all. There’s always something to say, and I’m interested in lead playtesters who are constantly processing each playtest session and are able to put their observations into words.

***

What questions would you ask to help you select the best testers?

Also read:

20 Comments on “Kickstarter Lesson #236: Selecting the Best Testers

  1. Take a popular game that you have played a lot, and break down exactly what you think works and doesn’t work about that game.

    That’s a freeform question designed to test how one gives feedback more precisely than question 1, which seems good, but it’s pretty obvious what the right answer is, and it doesn’t tell you if the person will actually respond with the specificity you are looking for.

    The question about what works and doesn’t work in a game will show you their writing and how they think – whether they resort to generalities and personal opinion, or specifically discuss how each mechanic might affects the player experience, in their view.

      1. Using the above question also helps you with question 5 does it not as you get an example of the type of written feedback you will receive.

  2. I read through them and did the test. I did not look at the correct answers on any of them until after I did it. My answers below. I got 5 out of 6. I got the 5th one wrong but the thing is, I enjoy writing, and I do write details.

    I think you have a problem with question 5. That might sound like sour grapes or I am just trying to get 6 out 6. But I don’t mind not getting it perfect and I enjoy writing. It’s easy to say I got 6 out of 6, if I was that kind of person, but I’m not. So for me question 5 was like “What do you least like? A) Ice cream, B) Chocolate, C) Cake”. The difference between them are insignificant (9.9, 9.8, 9.9). They all sound fun, so it’s kind of a trick question. I think question number 1 gives you the answer you are looking for, which is long-form feedback.

    1. C
    2. A
    3. B
    4. D
    5. B *
    6. C

    1. Anthony: I see what you’re saying. It’s a tricky question. The root of it is that I’ve had playtesters tell me straight up that they will provide detailed written feedback…and then they don’t. And it’s not just that they don’t–it’s clear from the way they write their feedback that they really, really don’t like to write. It’s not something they can self-identify–they’re excited about a game and want to test it–so I needed a question to help me identify something they’re not seeing in themselves.

      However, I see your point. If you’re the rare person who is equally willing to do any of those things, then I wouldn’t want to miss out on you as a playtester.

  3. It’s very interesting to see how closely this meshes up with what the Games Workshop Studio wanted out of their external playtesters while I was working with them!

  4. Jamey, do you know the Butterscotch Shenanigans video game studio? They are also from St. Louis. You’re from there right? Adam from BS wrote a good article about “GIVING USEFUL GAMEDEV FEEDBACK”. It gives examples of good and bad feedback. I found another thing in the article very useful from a creator’s viewpoint: Tell the playtester what kind of feedback you want (the scope of the feedback), because the feedback you want on the first 10 playtests are going to be different than the kind of feedback you want on the 700th playtest.

    https://www.bscotch.net/2015/11/giving-useful-gamedev-feedback/

    On and on question 5 again: Answer A takes zero effort so it kind of doesn’t exist for me. Which only leaves 2 options. C sounds super fun, interesting, and also requires almost zero effort, B is fun to me but requires a lot of effort, so picking C is kind of inefficient if the goal is to choose the “least”. The options are unbalanced if you think mathematically, and you have to pick one.

    1. I’m familiar with Butterscotch, but I haven’t read that article. I like the combination of (a) what happened, (b) why you think it’s a problem, and (c) how to make it happen again. I hadn’t thought of (a) and (c) being different things–maybe it’s more of a video game thing–but I could see that being helpful for board game feedback too.

  5. Yes, for a game designer we would probably describe (a) almost the same way we would describe (c). But with a playtester (a) could be a short story in a way they would tell a friend, and (c) would make them speak more like a game designer and go into greater detail.

  6. Definitely an interesting read. As Anthony talked about, it’s a weird concept to me to *not* go all out and be willing to do as much as possible for a playtest, or a review. The couple (smaller) reviews I’ve posted on BGG, including one for Scythe I did, made sure I fit them into that category. So it’s not something I had even thought about in terms of how I would find playtesters for the game I’m developing.

    I’ve noticed that the youtuber reviewers I tend to watch put out some long video discussions because it’s just more helpful when you’re looking at a new game to buy, and these aren’t even playtests – just reviews.

    Obviously, it’s difficult (but helpful!) for a group of people to rip your game-child apart. But I think something I, at least, can take away from this is that it’s not always enough to just say “Hey, this is a problem” and better feedback is “Hey, this is a problem because it causes this, and this is a way we found to make it better”.

    As me and my group start trying to get into playtesting, and might need playtesters soon, this is hugely helpful.

    1. Mike: Thanks for sharing your perspective. I would actually say that the “what” and “why” are much more important than “how to fix it”–that last part is often best left in the hands of the designer, especially since they’re gathering feedback from different people. So instead of “Hey this is a problem because it causes this, and this is a way we found to make it better,” I’d prefer to hear, “Hey, X happened, and it was frustrating to me because Y.”

        1. I agree, I prefer just to hear the problem and why. Just hearing a solution requires more work. I’m in playtesting with my board game and sometimes a playtester will tell my friend “you should do X” and it takes me a few days to understand why they said that, a few days to find the real problem. Usually the real problem is very loosely related to the proposed solution. Sometime solutions can be red herrings.

  7. Some great questions, Jamey! Really made me think about how I give feedback.

    I’ve been “lead playtester” in a few playtest groups for games for various publishers (and by “lead” I mean I organised the sessions, taught the rules, and fed-back responses to the designers/publishers; I didn’t necessarily “lead” the way the games were played or how the sessions progressed). Your questions certainly correspond to how I’ve given feedback to designers/publishers. I thought it worth noting here too though: On quite a few games now, we’ve had an organic feedback process using Slack for more conversational discussion; you build up a rapport with who you’re reporting back to, and with other playtesters from other groups who feedback comments as well, and generally you all support each other, bounce ideas off each other, discuss what went well, what went wrong, confusing rules, imbalances, etc. Letting playtesters from one playtest group answer rules queries or setup/gameplay issues from other groups I think gives even more feedback as to how everyone is interpreting certain rules and situations even when they *think* they got things right. It can prompt a lot more information out of people than simply filling in feedback forms.

    But back to your questions, which I had a few comments on:

    #2: I actually answered both (a) and (b). In the moment, I would note down the apparent lack of a rule, then I’d make a decision about what seemed the best way to go and carry on like that to finish the session. I’d then feed back that missing rule to you later with the rest of the feedback, to see how we could deal with it in the next session. I *think* that’s what you imply in (a), but it sounded a little like “well, we’ll just decide it must mean this and carry on” rather than noting the issue down. :)

    #3 was a difficult one. Honestly, I’d probably let them do (a), but I’d intentionally try to disrupt what they were doing by utilising some other strategy myself. Sometimes carrying over one “constant” from one game to another allows you to stress-test other areas to see how much of an effect they’ll have without having to intuit which bit of a change in play style affected things. Of course, if that player played the same way for say 6 straight games, then I’d certainly suggest they mix things up a bit. I guess if it was a 2-player game, it’s more imperative that players play differently each game. In say a 6 player game though, one player doing the same thing while the other 5 play differently than before wouldn’t cause much concern for me.

    #5: I agree with Anthony’s comments on this. Given the *choice*, i.e. if you personally specified we could do whichever one we’d prefer, I’d just film the playthrough as that’s easy, and you’d indicated we had the choice. Writing takes a lot more more time and effort and I’d prefer to spend that time doing more playtests. :) That would be my honest answer to the question. In reality, I’ve never submitted video or audio reports to designers/publishers [none I’ve done playtesting for ever asked for it]; it’s always been written (either an official session report on email/website, or conversational feedback on a Slack channel). A better question might be “which of these would you be willing to do” with multiple-answers.

    #6: I’d do my best to find *something* that felt confusing or frustrating to feed back. But I’d also assume a response-form would have a different field for what went well, so I wouldn’t put that in a “what went wrong” box. I *have* left blank responses in the past to some playtest forms which had that exact question, but only because I’d noted what went well in other places in the form, or because it was pretty far into the playtest process and there simply wasn’t anything negative to report for that session. Sometimes a session just goes super smoothly, and there’s nothing new to report (except “wow, that went smoothly!”).

    And oh my, I’ve typed a lot. Constructive feedback on your constructive feedback questions? I’ve clearly done far too much feedback on readability/style/grammar in prototype rulebooks! :)

    1. Thanks for sharing! I ran the Scythe playtest through something like Slack–lots of playtesters, lots of engagement between me and them, and it was very helpful. I might try that again on a future game.

  8. Currently in playtesting phase of a gamebook publication, I’m struck by the organisation with which how you structure this. Which has its benefits, particularly when working with large numbers of pt.s.

    However, I think there’s a place for intuitive responses too. A creator should be willing and able to plough through some ‘difficult’ responses in order to get feedback from marginal respondees – who may indicate the growth areas of the game audience anyway. Experienced gamers / designers with the ability to reflect on their gaming experience are only one part of an audience: if you hope to attract newbies / family gamers / young people to your project, I think it’s worth struggling with their feedback in the form they submit it – even to asking for clarifications / pursuing unanswered questions or giving free-form response.

    How you find and select /these/ sort of playtesters is a puzzler!

    1. Martin: I definitely agree that the feedback is often the start of a conversation instead of the end. And I agree that it’s good to get a variety of playtesters, not just experienced users. I want intuitive lead playtesters, but it’s also really helpful when they’re aware of when they’re having to use their intuition if something isn’t clearly described in the rules so they can report it to me.

Leave a Comment

© 2017 Stonemaier Games