We've had a couple of people contact us regarding the changes to Fivesecondtest.com. Not complaints, just suggestions. One user (Christopher) made a great response to my last post. I started to reply but then decided I'd just make a new post instead, as it is well worth the discussion!
You may or may not have noticed the last line of my original post, but it made reference to a "Karma" system. It is pretty much what Christopher described and it IS coming, we just haven't released the exact details yet!
A couple of suggestions have been made, which I'd like to respond to here.
1. "Why not just set a cap on results to a certain number"
We can't set "hard" caps on tests and responses. It may not be obvious, but the number of responses a test get varies with site traffic. Basically, the problem is that if we have a slow period and we have a hard cap we will end up with tests taking days and days to get all their results, or worse we may end up with no tests "waiting" in the system on a particularly busy day. We can't afford to have people going to go do a random test and be told there are none to do!
In actual fact, when are traffic spikes it is often either just people DOING tests or just people CREATING tests and this makes coming up with a "cap" nigh on impossible. The number of "expected" results per test can vary from 9 responses all the way up to 20 or more (with premium tests being a multiple of that).
Now, what used to happen was that paid tests "expected" results were based on the average usage for the week. This means that variations to that number would occur slowly over time. When we saw a spike in traffic, rather than paid users reaping the benefits, free tests were "absorbing" that extra traffic (because when there are no paid tests awaiting results, we just grab the lowest free tests). We'd get free tests getting 20 or more results and bronze test (worked out on the weekly average) only getting 22 or 23.
What we're now doing is making sure paid tests are absorbing that extra traffic, rather than free tests....hence why free tests are now fixed at 5 responses. When we get a spike in traffic, those extra results go to people have handed over their hard-earned cash. Of course, we still need to smooth the numbers a bit so that we have a reasonably consistent number from day to day, so to a degree free tests still absorb some of the traffic spike (at the moment most free tests are getting 7 or 8 results)....just not as much as they used to.
2. "Why don't you reward people for doing tests?"
The karma system (which IS coming soon) will work pretty much how Christopher described it (i.e do X tests and get a free token). But it's not as easy at it seems. At the moment tests have very little "noise" in their results. People are doing tests because they want to help someone else out. By relying on the good nature of people, we ensure that test results are good quality. People are filling out tests because they want to help, not because there is a reward for doing so.
The concern is that if we start giving away tokens for responses, we will be encouraging "noise" (i.e. people doing tests purely to get credits, not to be helpful). So what need to do (for memory tests) is allow test owners to vet their responses. They will be able to discard worthless responses and "reward" very useful responses. The result isn't then just about how many tests you did, but also how useful your comments are. The problem here, however, is validating "useful" clicks in the click test. We are yet to come up with a good way to reward "good" clicks and deter "bad" clicks....how do you even tell the difference? That is the real stumbling block to the Karma system.
3. "can I just buy a subscription?"
We'd love for you to give us more money! But seriously, we're not sure how a subscription results in better value for our users. Subscription models are great for the vendor, but I'm not convinced they're better for the end user. The big upside for users, of course, is that they don't have to deal with tokens (which we admit can be a little confusing!). The downside though is that having a certain number of tests available per work or month isn't really how we're seeing people work. I don't ever want a situation where someone has bought 4 tests per month, and all of a sudden needs a 5th, but that requires upgrading to the next plan.
For me personally, the pay for what you use method is much better value for money. The vast majority of our sales are from people who buy tokens in bulk, and use them when they need them. If you need more, buy some more. We've just now included larger token packs to make this a whole lot easier to do this. Of course, if people keep nagging for subscriptions...we will add them! So keep your feedback coming!
4. Last but not least, "can I just see English/French/Chinese/German tests".
Another tricksie problem! There are two ways to look at this problem. Test owners don't gain much value from people doing their test if they can't read the writing in it, and similarly users can get frustrated doing tests in other languages. The trick here is marrying the two together. We want to give the test owner the ability to exclude anyone who doesn't "accept" that language, but we also want end users to be able skip all tests that don't match their language choice. Doesn't sound hard does it?
The problem arises in that most of our test "doers" are anonymous. Beyond checking their browser language we don't know what they do or don't understand. We also don't want to trouble users while registering with too many demographic questions. So the solution at this point is to do a little of both. We'll be adding the ability for registered users to submit their demographic information (if they choose!), and we'll be adding the ability for test owners to exclude all but certain groups from doing their tests. We'll also then have the ability for test owners to filter on that demographic information. We then just need to sort out the anonymous users....
Well, I hope that answers some of the questions floating around! We are still actively making changes, and so your feedback definitely DOES help us decide what to do next. We're both pretty busy on other projects, so changes don't happen quite as quickly as we'd like, but they ARE coming.
Thanks again for all your feedback and support!