Questions, Answers and the "Karma" system

We've had a couple of people contact us regarding the changes to Not complaints, just suggestions. One user (Christopher) made a great response to my last post. I started to reply but then decided I'd just make a new post instead, as it is well worth the discussion!

You may or may not have noticed the last line of my original post, but it made reference to a "Karma" system. It is pretty much what Christopher described and it IS coming, we just haven't released the exact details yet!

A couple of suggestions have been made, which I'd like to respond to here. 

1. "Why not just set a cap on results to a certain number"

We can't set "hard" caps on tests and responses. It may not be obvious, but the number of responses a test get varies with site traffic. Basically, the problem is that if we have a slow period and we have a hard cap we will end up with tests taking days and days to get all their results, or worse we may end up with no tests "waiting" in the system on a particularly busy day. We can't afford to have people going to go do a random test and be told there are none to do!

In actual fact, when are traffic spikes it is often either just people DOING tests or just people CREATING tests and this makes coming up with a "cap" nigh on impossible. The number of "expected" results per test can vary from 9 responses all the way up to 20 or more (with premium tests being a multiple of that). 

Now, what used to happen was that paid tests "expected" results were based on the average usage for the week. This means that variations to that number would occur slowly over time. When we saw a spike in traffic, rather than paid users reaping the benefits, free tests were "absorbing" that extra traffic (because when there are no paid tests awaiting results, we just grab the lowest free tests). We'd get free tests getting 20 or more results and bronze test (worked out on the weekly average) only getting 22 or 23. 

What we're now doing is making sure paid tests are absorbing that extra traffic, rather than free tests....hence why free tests are now fixed at 5 responses. When we get a spike in traffic, those extra results go to people have handed over their hard-earned cash. Of course, we still need to smooth the numbers a bit so that we have a reasonably consistent number from day to day, so to a degree free tests still absorb some of the traffic spike (at the moment most free tests are getting 7 or 8 results)....just not as much as they used to.

2. "Why don't you reward people for doing tests?"

The karma system (which IS coming soon) will work pretty much how Christopher described it (i.e do X tests and get a free token). But it's not as easy at it seems. At the moment tests have very little "noise" in their results. People are doing tests because they want to help someone else out. By relying on the good nature of people, we ensure that test results are good quality. People are filling out tests because they want to help, not because there is a reward for doing so.

The concern is that if we start giving away tokens for responses, we will be encouraging "noise" (i.e. people doing tests purely to get credits, not to be helpful). So what need to do (for memory tests) is allow test owners to vet their responses. They will be able to discard worthless responses and "reward" very useful responses. The result isn't then just about how many tests you did, but also how useful your comments are.  The problem here, however, is validating "useful" clicks in the click test. We are yet to come up with a good way to reward "good" clicks and deter "bad" do you even tell the difference? That is the real stumbling block to the Karma system.

3. "can I just buy a subscription?"

We'd love for you to give us more money! But seriously, we're not sure how a subscription results in better value for our users. Subscription models are great for the vendor, but I'm not convinced they're better for the end user. The big upside for users, of course, is that they don't have to deal with tokens (which we admit can be a little confusing!). The downside though is that having a certain number of tests available per work or month isn't really how we're seeing people work. I don't ever want a situation where someone has bought 4 tests per month, and all of a sudden needs a 5th, but that requires upgrading to the next plan. 

For me personally, the pay for what you use method is much better value for money. The vast majority of our sales are from people who buy tokens in bulk, and use them when they need them. If you need more, buy some more. We've just now included larger token packs to make this a whole lot easier to do this. Of course, if people keep nagging for subscriptions...we will add them! So keep your feedback coming!

4. Last but not least, "can I just see English/French/Chinese/German tests".

Another tricksie problem! There are two ways to look at this problem. Test owners don't gain much value from people doing their test if they can't read the writing in it, and similarly users can get frustrated doing tests in other languages. The trick here is marrying the two together. We want to give the test owner the ability to exclude anyone who doesn't "accept" that language, but we also want end users to be able skip all tests that don't match their language choice. Doesn't sound hard does it?

The problem arises in that most of our test "doers" are anonymous. Beyond checking their browser language we don't know what they do or don't understand. We also don't want to trouble users while registering with too many demographic questions. So the solution at this point is to do a little of both. We'll be adding the ability for registered users to submit their demographic information (if they choose!), and we'll be adding the ability for test owners to exclude all but certain groups from doing their tests. We'll also then have the ability for test owners to filter on that demographic information. We then just need to sort out the anonymous users....

Well, I hope that answers some of the questions floating around! We are still actively making changes, and so your feedback definitely DOES help us decide what to do next. We're both pretty busy on other projects, so changes don't happen quite as quickly as we'd like, but they ARE coming.

Thanks again for all your feedback and support!

6 responses
I think the problems you identify with the Karma system aren't, really. Sure, a few jerks will click click click through something just to get a token. But the majority of your users will realize that if they provide valuable feedback as a rule, they'll GET valuable feedback as a rule. We also want to help others as we'd like to be helped - the tokens are a reward, yes, but they're payment for work that we all, as designers, take very seriously.

So instead of having a "I liked this person's clicks, so I'll award them a credit" kind of a thing (what if the design stinks and the designer is ticked that the majority of users didn't get the "right" answers?), I'd suggest a reporting system instead, where you can "flag" a dud response - if a user gets enough flags, then somebody takes a look at what's going on.

But I predict that this would be relatively rare, though. Maybe I'm an optimist :)

Hi Renee,

I quite hope you're correct! :)

We're looking at a few ways to encourage positive activity. Being able to remove "dud" comments is definitely one thing we'll be doing. We're also looking at encouraging "regular" participation over "quantity" of participation. The thought being that someone who does 5 tests a day for 20 days is arguably more valuable to the community than someone who does 60 in one day and never does any again.

At the end of the day, we just need to find a balance between rewarding those that are doing a service to the community, and ensuring it doesn't get abused. I think we're getting close though!

Cheers, and thanks for your input.

I love your service very much! But I often miss one thing that I'd really appreciate to see on your site: Next to memory and click tests it would be great and a big benefit, if we could define a task for a click test (lets say "You want to find a list of all books in the shop") and then ask to click at the "link", where the tester assumes to pursue that target.

Is there any chance to see such a feature on your site in the future?

(I hope my suggestion goes to the right place)

Hi! Glad you like our site!

Hrm... not sure if I should answer your question :)

The short answer is, No, we will not be adding this feature to

The long answer is, Yes, but it will be a new test under a new banner. The main reason is that to carry out a task, a user needs more than five seconds! I can't tell you any of the details as yet, but it will do precisely what you're hoping for! We're in the process of building this new shiny product, and hope to launch a Beta version some time in March!

Keep your eyes here for more info in the next few weeks!!

Thanks for your support!

I just submitted a test. How do i get feedback if you don't know my email address? Thanks.
Hi David,

If you were not a registered user, you should bookmark your results page. If you're on the same computer you created the test on, you could try to go to as you will have a cookie set with your test details.

FYI, we'll be getting rid of the ability to create anonymous tests soon. So I'd recommend registering.

Hope that helps.