Survata
We’ve been using a tool at USV recently that I like. It is called Survata and it allows to you to create a survey and then target it at whatever number of completes you want. You can target it to respondents in 17 countries “by age, gender, geography, and custom attributes.”
It is helpful for us to get a sense of what is going on in a market quickly. We generally go for thousands of completes and we get results within three to seven days. We have been paying between $1 and $2 per complete which, for us, is not a lot to get some answers quickly.
I am sure there are many services out there that do something like this. It’s not particularly difficult to offer a service like this. But the Survata user experience is simple, easy, fast, and affordable. We like it and are using frequently. I thought I’d share that with all of you.
Comments (Archived):
I was going to ask on quality of data, but will follow others to get that answer. My question is maybe more selfish than market interest. What if you are using surveys to target specific countries and you don’t want to canvass an area? Can you customize per geography? I guess I can answer this by digging around there myself.
In most countries you can target by region, city, and sometimes postal code. In the US, you have a lot of geo-targeting options.
In our Basic Survey Package, we offer targeting in 5 countries, including region, state, city, and DMA. Our Advanced Survey Package offers respondents in 70 countries, in any language. You also have the option of using screening questions to get more details on where a respondent is located, available in both packages.
Interesting that their model is to pay the intermediary (publishers) who unlock content for survey takers. I’ve been using another variation on this service where they pay the consumer of the survey directly. I get paid 50 cents for answering a Q. See screenshot:
you get paid $.50 for answering a question and this is indicative of a good process, of reliable information?you are an analysts by nature william.why ever would you use a survey to gather information and how would you value the responses?
I was trying it for a friend. I have no idea how useful it is. I get asked a Q once per month at the most, and if you don’t answer within 30 mins, it passes you.I am not judging if it’s good or bad. It’s interesting & I’ve only managed to catch the answering window twice, so I made a $1 which PayPal will take 30 cents from ironically.For me, this is just a fun experiment. Of course, lots of flaws in this process, but some companies are desperate for information.
Of course.I used surveys extensively for a decade so understand them well.While this one is clever in its implementation it is still the same thing that’s been done for a couple of decades.Not saying that its not clever, just that this approach to collecting opinions seems so broken. So broken in fact to me that its not worth doing in the old way
Is Veespo the new way? Tell us more 😉
David answered above William.It is ‘a’ new way that works. From the ground up built to do this differently.I’m a believer big time. Market is telling us the same.
Not a believer in surveys since I stopped using them as a marketing tool in the mid 90s.A last century technology for a pre web mindset honestly.We get bombarded with requests to do them. We get offered pittance to answer questions which don’t allow us to express any true opinion.Seems broken to me in so many ways. A disconnect between the immense need we have to understand markets and our customers thoughts and rigid tools for collecting that info.Square peg, round hole.
Amen, Arnaldo.Traditional surveys are a burden forced upon reluctant users — only companies like them.
David, you are in this business. How does Veespo do this?
We provide a SAS layer which allows anyone to build any type of opinion collection service. We’re not limited to market research; our platform can be used for live TV, sporting events, conferences, customer feedback, etc.What makes our platform different is that Veespo does not burden the user. The experience is much closer to social commenting, but we are still able to aggregate the responses and so provide all the analytic tools you would expect from surveys.
Interesting. Thanks!
It’s a pleasure William, thanks for asking.
When will it be available in North America?
At the moment we’re building a portfolio of flagship implementations here in Italy (that’s why the website is only in Italian). Over the next few months we’ll open it up to international users and we’ll transition to our fancy new English website….
Thanks, David. I looked at Veespo about 2 years ago. It was something I compared with Audipress, Sinottica and other market research providers in Italy.I speak and read Italian.
Should add to David’s comment that while the clients are in Italy, many are large global companies who service clients everywhere.One of the key pieces is that the platform can be used in any language automatically enabling feedback cross culture, cross language.Platform agnostic so works on mobile or web.
Arnold, totally agree with you. Please see my comment.It is totally ANACHRONISTIC technology and broken at the root-level of Rensis Likert (where the 5-stars comes from) and Descartes-Pascal-Bayes structures for Probability.In fact, I wrote all about survey flaws dating back to the invention of advertising itself in the 1920s and 1930s in the published patent of my system.
Customer feedback is everything.Check out Veespo as a company that is tackling feedback from the ground up on a global, multi language, digital platform.
And one horrible use case of the survey is the “360-review” of someone’s performance at work. Reviews like this can be useful, but you need a tuning fork (nice analogy, @twaintwain:disqus ) and not a, well, hammer.
Thanks, Anne. Sometimes I use the analogy of “It’s like using diving goggles instead of short/long-sighted glasses on land.”I wear glasses and we know that seeing 360, of and in itself, isn’t adequate. We need our glass lens to be 20/20.Probability & Stats & 5-stars and those 360 reviews are like diving googles.Remember Hugh Grant in ‘Notting Hill’, lol.
Interesting. The business school program that I participated in (and then worked for) was predominately market research using surveys, focus groups, and interviews. By far the one on one interviews were the most useful tool, however.
Surveys are broken by design–and design that was built on very old technology.Want to know what people really think on the web, its all in the social gestures.What to aggregate that into data–kind of a gaggle of analogue blurts–good luck.The key is flexibility on the input side that is personalizable and digital data on the output side so it it useful.Fascinating stuff.
We get bombarded with requests to do them. We get offered pittance to answer questions which don’t allow us to express any true opinion.Agree and it’s almost comical. I received a recent 10 page survey with $1 dollar in the envelope after a recent very expensive car purchase. I found it almost comical that they would think that a buyer of that car would take the time to answer any survey no less a very long survey. (I mean a postcard with 10 questions might have been fine and by the way skip the “dollar”).We get offered pittance to answer questions which don’t allow us to express any true opinion.That for me is really also part of the issue. I love to give my opinion on things. What I hate is to have some totally “on the clock” person ask me a bunch of rote robotic questions to me it’s a complete waste of time.For example I’m sure if a college student (with a project) called you and wanted to discuss some issue and have a short conversation you would take the time to help. But not if the same college student was just doing a survey because they were being paid to ask some questions and gather answers.
Great points. We agree, which is why we don’t power the sorts of surveys you are referencing. While we’ll provide a tool to survey your own customers in addition to our market research product, it isn’t a core product, and isn’t usually something our clients ask for. We don’t pay our respondents, they answer questions to unlock content or experiences in our publisher network. Our SurveyWall consists of a max of 8 questions (6 questions and up to 2 screeners), and takes roughly the same amount of time as a 30 second video ad to complete. We’ve found that by keeping the length of time required to complete the survey limited to the length of a familiar ad type, our responses tend to be high quality, and the amount of cleaning required by us isn’t prohibitively high.
Voilà and Da Vinci showed us back in 1490 all about square pegs and round holes.In the square, the man is static and rigid.In the sphere, he’s free and flying.Both states of Man exist, in synch. He is both 0 and 1.
Ok so this is a space I’ve done comprehensive competitive analysis for because of my Startup #1 (Senseus collects and analyses data). Startup #2 is Sygnyls. Yes, I focus on systems that produce SIGNAL >>> noise and enable us to make sense of data intelligence.Anyway, I’m not a fan of existing survey systems — whether survata, typeform (used by Startup Grind), Google surveys (including Polar polls), Buzzfeed, any number of mobile surveys including Audience Science, Mixpanel and others.The reason isn’t simply the form factor and user interaction aspects of them.My objection to them goes deeper and includes the need for first principles, root-level innovation.It has to do with 5-star ratings being a flawed and redundant legacy approach —TechCrunch, Sept 2009: YouTube realizes 5-star ratings are useless:* http://techcrunch.com/2009/…It’s a known persistent problem which 99% of developers, data engineers and mathematicians haven’t been able to solve. In fact there are several threads about this on Stackoverflow and Quora.It also has to do with how the data collected lends itself to fundamental flaws with Probability & Statistics as tools for measuring human thinking (including subjective biases), language, decision-making, choices etc.See MS’s stack ranking problems which some people attribute to being why it experiences its “Lost Decade”. Stack ranking involved trying to fit people into the Normal Distribution curve:”At the center of the cultural problems was a management system called “stack ranking.” Every current and former Microsoft employee I interviewed—every one—cited stack ranking as the most destructive process inside of Microsoft, something that drove out untold numbers of employees. The system—also referred to as “the performance model,” “the bell curve,” or just “the employee review”—has, with certain variations over the years, worked like this: every unit was forced to declare a certain percentage of employees as top performers, then good performers, then average, then below average, then poor.”People would then be assigned one of three grades: 4.0 (Above Average), 3.5 (Average), and 3.0 (Below Average). The very rare 4.5 got you a set of steak knives; a 2.5 meant you were fired (more or less). I’m not sure if Alan Turing himself could have gotten a 5.0.”* http://www.vanityfair.com/n…Sure, surveying tools are useful but it’s a bit like using a knife when what we really need to invent is a tuning fork.No one who works with me will ever be subjected to and biased against by Probability Distribution Curve stack ranking or 5-star ratings…….Because I invented a much more considerate and intelligent system.
Interesting – I *think* performance/employee review internally is a bit different than market research and data collection though.In the later case, often the goal behind what you are after really is all about identifying the ‘average’ as well as the ‘edges’ and you almost never want to apply it to specific individuals (this is why, generally, the larger and wider the sample pool the better).Sounds like what you have invented though would be *way* more useful/interesting for reviews and situations where you really are trying to focus on individuals (and how they can grow and better contribute to the whole of something).Side Note/Question: Can you share more about the ‘how’ and ‘why’ your method works better?
At the core (in terms of mathematical tools applied), internal employee review and external market research methods share the same problem.It’s not only about the variance in objectives between the two or issues wrt the “means to the end” (e.g. which probability & statistical tools we use to clean-scrub the data, check for its veracity and independence and sample representation, do we use linear regression or gradient descent to pinpoint multivariate saddle points or Nth derivative with latency assumptions?) or about the outcome and the interpretation of the data (what market researchers call “actionable insights”) that needs to be re-thought and re-engineered…It’s understanding more about the very nature of our human minds and what we know/don’t know about how we see, measure and model the world around us.What happened was a fork in the road of our tool invention history (pls see graphic).Probability was invented in 1654 for the narrow objective of measuring the stochasticity of dice (which — unlike us — don’t think, don’t have languages, don’t have subjective biases that inform decision-making and don’t adapt).For centuries Probability was indeed very useful for being a tool that improved production efficiencies; it enabled the Industrial Revolution when mechanization appeared in manufacturing.However, it has mistakenly been applied BEYOND ITS REMIT to measure the human mind and our behaviors. It got adopted by psychology when it emerged as a field — hence Rensis Likerts “strongly disagree…….strongly agree” scale which is the basis for market research surveys, 5-stars and stack ranking.Now, we aren’t dice or machines. We have language. We have values. We have subjective biases which Probability wasn’t and isn’t designed to handle.I’d argue Probability is the root source of why, in Machine Intelligence, they (Google, Baidu, FB, IBM Watson et al) haven’t been able to program the machines to understand language and its meanings. All of these techco’s have admitted we need to invent new language models and tools.Language and meaning are subjective. Subjectivity is factored into, for example, the subjunctive tenses of non-English languages. This is why translation software doesn’t actually understand what we’re saying because translation software is all probabilistic correlations.Notably, probability was designed to measure stochasticity and parameterize by “fuzzy logic”. It wasn’t designed to measure subjectivity and to parameterize by feelings.So…even before we architect the code stack we have to go back to the very roots of where the fork in human tool invention happened.We discover Da Vinci shone the light for the way forward and how to build an integrated system.In terms of a stack, it has to be modular and polyglot in design.In terms of the code language, well…….it has to include Quantum Mechanics notation for semantic-sentiment structures.In terms of user interaction, the UX design has to have constraints but not the same ones as existing survey systems with their “tick the box” / “fill in the free text box and we’ll do text-mining on it” / “use this slider to select how much – how little you agree with this statement”.It has to be first principles, root-level invention.Pretty much akin to what Bitcoin is doing for 300+ years of “How money and finance works” architecture.Now, the thing is…There will be resistance to any tool that seeks to be different from yet complementary to Probability.However, if we mean to achieve such noble objectives as:(1.) The machines actually being able to qualify and quantify data value, and help us make better decisions — e.g. reduce risks of future global financial crisis.(2.) Machines being able to consider and not simply think logically and optimize. Consideration means they’ll care about us and not become ‘Terminator’-types who just think about optimizing their survival at the expense of human life and values.(3.) Increase diversity in tech…Then first principles, root-level invention is a necessity we should do now.
Good stuff…but sometimes probability (of action) is all you’re really looking for…though I’m probably wrong about that…
Thanks :*). Global society’s become very focused on the number outcomes.How we get there has, in many ways, lost its voice and visibility.Fred wrote ‘Valuation as a scorecard’ yesterday.Is that end number outcome (measured action point) the only thing which matters? If we correlate those with Probability will the numbers give us everything we need to know and feel?Or is it equally valuable to be able to understand and measure, “WHY do people love this product / company / relationship?”Can Probability measure the Why’s?If it can, then why do the scientists all agree that Probability and Statistics is about correlation not causation?
True enough that one should not conflate correlation with causation but it is still worth noting that they tend to frequent the same neighbourhoods.
Oh they’re more than frequenting the same neighborhood.Da Vinci showed us the way towards their integrated model back in 1490 with ‘Vitruvian Man’ before Pascal & Descartes forked us down the path of Probability.Descartes, in particular, is to “blame” for the wrong-headed way we’ve separated head, heart and soul and mind-body into Cartesian points for correlation rather than modeling us as dynamic wave continuum.
Interesting. Then the leaders in Behavioral Econ that I have read are using bad framing to build research off of.
The research that Daniel Kahneman and Amos Tversky did in the 1970s is the basis for a lot of the Quantified Self, AB testing, behavioral heuristics measured by today’s startups as well as in economics.Now, when we read into Kahneman’s and others’ pointers about the conjunction fallacy in Probability and when we’re aware of this 2014 neuroscience research from Harvard Medical School on ‘The Perceptions of Probability’ which examines whether our minds think probabilistically or applies a whole other type of algorithmic framework…….It could indeed be the case that amiss framing about our intelligence, decision-making and consumption choices has been at play.Market surveys, economic models, 5-star rating systems, probability-based Machine Intelligence et al were created and layer-by-layer built up at a time before we had Neuroscience and DNA as tool constructs for understanding ourselves better.
So…do we spend another 30 years with amiss framing or do we innovate another framework?
Harvard Medical School on ‘The Perceptions of Probability’Sounds like a good read !It does beg the question:How does one separate the deeply nested subjective entanglements inherent in personal linguistic perceptions from the probabilistic substrate of human mind mechanics ?That smells of self-refferential confusion ?
Professor Charles Randy Gallistel’s 2014 talk at Harvard’s Distinguished Lecture on “Perceptions of Probability* and how measuring behavioral actions tells us little about why we think, decide and do what we do. It has implications for Machine Intelligence structures and our understanding and models for neurons and neural networks.His first slide shows:PERCEPTION OF THE ABSTRACT===========================* We perceive distance, duration, rate, probability etc — the abstract* So do rats and mice* But these percepts have no qualia* And no readily identifiable stimulus* https://www.youtube.com/wat…Then watch Rick Born of Harvard Medical School’s presentation.
Thanks for the link !
Please also watch Professor Charles Randy Gallistel on behavioral heuristics:* https://www.youtube.com/wat…
In terms of a stack, it has to be modular and polyglot in design.Like Gödel’s incompleteness theorems point out, no single formal system can give a universally complete modelling perspective due to its self-referential internal-consistancy construction-constraints.Complex/living-systems seem to solve that problem via distributed redundancy of form/function that all compete for inclusion into statistically integrative outcomes.(non-trivial-causal-spread)I can’t even pretend to understand the math here but it smells to me like “two-mint-in-one” ?
.Performance appraisal is an aspirational endeavor with an employee with whom the company has a very substantial investment.Market research is a shot in the dark with a “meh” customer/potential customer.While sampling techniques may be similar — not agreeing to that notion, mind you — the skin in the game is hugely different.Performance appraisal is a process while market research may be a one time endeavor.This is a huge difference in my book and it troubles me that smart people would not see the obvious differences.This is like the difference between checking the plane’s paint job and ensuring the fuel load is adequate.JLMwww.themusingsofthebigredca…
In performance appraisal we’re typically assessed by:* line manager who works directly with us* peer colleagues (one in our team, others from other teams)* internal clients we’ve delivered for* external clients we’ve delivered forThey have CONTEXT of us personally. That context being objective and subjective — in terms of their assessment of our problem-solving, team work, communication skills etc. and how we reciprocate and confirm/modulate their assessment of us in our self-assessment.Those assessments by third parties being informed by and entangled with their implicit EMOTIONAL experiences with us, and our emotions with them.In standard market research surveys the emotions are abstracted out.So not only is a one-off market survey a mere snapshot that’s not necessarily representative of us, it also doesn’t take into account our emotional interactions.In both performance appraisal and market research surveys, the implicit emotion factors haven’t been measured or factored in explicitly.It may / may not be contributing source factors to how men and women are assessed and promoted differently which is causing the angst in SV about inclusion and diversity.
.I agree with much that you say particularly about the presence or absence of emotion in performance appraisal.A great performance appraisal system has at its core the contribution of the individual to the business plan of the company with a Vision, Mission, Strategy, Tactics, OBJECTIVES, Values and Culture perspective.In this manner, the individual objectives of an employee are the yardstick that is initially held up to evaluate performance. This assessment is devoid of emotions.I think many companies miss this and devolve to an emotional standard — as you have articulated well — which really has nothing to do with whether the company’s mission is being achieved.On the other hand, I would argue that a focus group is an effort to delve into the emotions of a real or perspective customer rather than a bloodless exercise.If you were working with a group of Apple customers in a focus group on a new product and they failed to rise to the expected level of fandom, you would know the absence of that emotion could foreshadow disaster.It strikes me that you are overthinking the matter and trying to reduce things to a level of certainty and mathematical precision that simply cannot be done.Don’t get me wrong, it is a thoughtful and interesting discussion and your insights are broadening and powerful. It is cold fusion v the wooden match.JLMwww.themusingsofthebigredca…
JLM, I’m borrowing this one from you too: “It is cold fusion v the wooden match.”So a lot of thinking did happen before I went to see my former Head of Maths at university and said, “Probability doesn’t work for us anymore. So I’m making another system. Something based on perceptions and DNA-y.”He gave me his “What are you up to now?!!!” look which I remember from when I was a teen and in his classes. He taught…Probability and Statistics, and I got 99% in his exams. He was also my personal tutor so aware of my quirks of thinking.Anyway, fast forward to Oct 2014 and he sent me an email after seeing my system: “My instinct is that developing qualitative techniques for analyzing things that aren’t really quantitative must be the right way to go.”So he’d done a 360 turn from rational skepticism to agreeing with me.Wrt “Reduce things to a level of certainty and mathematical precision simply cannot be done”, reducing our emotions to a probability (sad = 0.45, happy = 0.48, excited = 0.46) is how they currently do Machine Learning as well as how W3C structures emotions for html.This is the existing model that’s supposed to be some type of replication / reflection of our intelligence and it’s a model I fundamentally disagree with.The epiphany I had was this…….The tool to enable and increase the precision is NOT from mathematics or Quantum Physics.It’s from biochemistry and requires the invention of a tool system that can convert emotions (which is a biochemical phenomena) into a mathematical and code coherent tool that isn’t a Probability function.AND the inventor has to design a UX for data collection that even a 2 year old can use.The system then generating data which the PhDs in Risk Arbitrageurs, rocket Data Science and Machine Intelligence-robotics can run all their complex N-th derivative, probabilistic Hidden Markovs on to their heart’s content.It can be done, :*). I know because I made it.
.I do agree that much that is qualitative can be converted to quantitative with the caution that it is only one input and should never be allowed to stand alone and be the only bit of data with which to make a decision.If you have enough data, the analysis begins to regress to something that is truly predictable and predictive.The most important thing is to think — not to make a knee jerk decision but to allow it to be one more piece of data that may provide the keystone to the arch.JLMwww.themusingsofthebigredca…
A “probability function” is an approximation-tool that stands in for a “mathematical/code coherent tool” in situations that are simply to complex/self-referential to be perceptually penetrated by reliable metrics.(Heisenberg’s-uncertainty-principle/measurement-interference-principle)Even biochemical based intelligence systems would seem to bottom out as statistically causal expressions/applications of even more fundamental quantum-mechanics substrate uncertainties.Can you share any overview of how you have managed to penetrate that mystically self-refferencial measurement constraint ?
I knew either you or SigmaAlgebra would ask the Heisenberg question, :*).So Peter Thiel wrote this in my copy of ‘From Zero to One’: “For Twain — never compete.” He had no context of me — other than doing a double-take when he heard my name’s Twain.In fact, I don’t compete. I try to integrate (Art + Science, male + female, East + West… that type of thing).Anyway, when I finally read Thiel’s book in April and saw this graphic inverting the classical Probability Distribution Curve I LOLed.As we can see…Thiel’s inversion is still only 1/2 of a wave and a fraction of the planes (the graphic is 2D on x-y).Heisenberg’s penetrable by resolving the superposition paradox in Schrödinger’s wave equation and spotting that Probability can’t possibly solve it because…In Bayes’ binomial path, the state outcome is deterministic: cat is dead = 0 OR the cat is alive = 1 with a probability likelihood between the two states (0.1, 0.2, 0.3 etc).Schrödinger and Einstein made a case for the cat being both dead and alive, so 0 and 1 exist simultaneously.Probability can’t do this so it falls apart as a tool at the Quantum and atomic Intelligence level.As does Turing’s version of machine intelligence…
Probability falls apart as a tool at the Quantum and atomic Intelligence level.As a tool probability functions as an approximation-tool. When such a approximation tool is pushed too far down into the realm of quantum mechanics it flips into its opposite and becomes an impediment.I just wonder, does it fall apart or come full circle to the phenomenological-limit of natural/mathematical languages to externally model a system of which it is an integral self-referential subset ?Or as Allan Watts might say”A knife can’t cut itself”PSYou are hurting my old man brain. All your math is way over my head. I’m just having fun responding on a somewhat simplistic philosophic plane.
Ah, so that’s precisely the point. The very limit of mathematics as a tool itself.Is it, as a language, an adequate enough one for interpreting other languages (English, Chinese, Spanish, the interpersonal language between other cultures) and for qualifying — not simply quantifying — phenomena, including our decision-making and behaviors?Now, Von Neumann in his last book wrote: “When we talk mathematics, we may be discussing a secondary language built on the primary language of the nervous system.”As with Turing’s model for AI, the Von Neumann model for computing was frameworked at a time BEFORE Neuroscience and the discovery of DNA (and particularly the research that suggests up to 80% of our intelligence is attributable to the X-chromosome.They were frameworked according to the expedient tools of THEIR time: Probability.If Von Neumann’s supposition is right and mathematics is a secondary language then it follows logically that Probability is also secondary as a language.Moreover, if the primary language is in the nervous system then it needs a BIOCHEMICALLY-based tool to modulate mathematics as a language.Then the phenomenology would be closer to coherency.Philosophy aside, there is actually a practical application system coded.It simply does EVERYTHING DIFFERENTLY from the existing approaches to opinion surveys and trying to understand people and our data, quantitatively and qualitatively.
The problem for me w/ performance appraisals isn’t necessarily their 5-point scaling, it’s more their generic use and design, particularly in large size orgs. An evaluative process is good for managing employee performance, expectations, growth, etc., but greater customization by job function is needed, although few companies actually invest the time to do so. The appraisal form should be designed to dovetail w/ the employee’s stated job description.
Yes, you’re right. Generic use and design is a big problem.The same template gets sent to an Accountant as to a Field Sales person as to the Receptionist in the lobby and their milestones are very different: the first’s objective is to manage (sometimes cut) costs, the second’s is to ramp up and book revenues, and the third is to be the welcoming face visitors see first and who navigates them to the right meeting room.Very different roles and objectives.The other issue, of course, is that reviews typically happen once (maybe twice) a year and people have short as well as (sometimes) unreliable memories of what milestones someone may/may not have delivered.The one-size-fits-all, once-a-year snapshot of employee performance is really illogical.The irony being that in b-school they tell us to take continuous measurements (the video rather than the snapshot) and yet in HR they don’t implement this.
For performance reviews check out juvodhr.com. Custom. Takes emotion out of it and gives you info/feedback when everyone needs it.
I think that’s the general point I was making…at least it was in my own head 😉
I love Twain. but if a gal can have too much going on in the ol’ cranium, it may be her.
:*), James. Hey so my Chief Scientist (whose little gray cells also work over-time; he’s got a PhD in AI) in Startup #2 LOLs all the time. Yesterday he wrote, “Holy shit, Twain! Nice work!” We’re making something really fun involving sensors that’s hard to do.We’re the first team in the world to do it and Disney may incubate our idea.Meanwhile, to change defunct and anachronistic systems like surveys and mathematical tools that no longer work for us, we have to critically assess and understand EVERY facet of it.We do this at Big Picture and nanoscopic levels.We zoom in and zoom out. We see the higher level strategy and the details. We try to connect the dots and pinpoint root causes.Then we can do what Elon Musk advocates: invent by first principles.Let’s imagine existing surveys don’t exist. We want to understand people and their meaning and intent.How does that happen? How do people think and decide? Through mathematics or through language and sensory cues? The assumption ever since John Graunt collected mortality rates as data is that the mathematics gives the data structure and meaning.What if that’s amiss? What if the meaning of people isn’t in the mathematics — regardless of the Probabilities, percentages, #votes up, 1st in ranking, $1 compared with $100 per survey?What if it’s amiss because whilst mathematics is indeed a language to quantify, it’s not the language to qualify? How do we make it a language that can quantify and qualify?In the data collection (surveying) techniques we’ve inherited, is the inadequacy of the tools something we’re even aware of? Or do we just use them because we don’t have the imagination or guts to make better tools and systems?In cases where about 300 years of an encumbent system is being transformed, a LOT of thinking and doing gets done.It’s true for Bitcoin. It’s true for Tesla Powercell. It’s true for data collection and understanding that data.Arnold pointed out that surveys are “broken in so many ways” and he stopped using them in mid-1990s.Now, Fred noted, “It’s not particularly difficult to do a service like this” (survata) and he’s right. If it’s not difficult then what’s the USP or proprietary IP that would make survey systems like survata able to build a monopoly position similar to what Thiel believes Google has?Another question is, “Do we continue to do things (which we know to be broken ) in the same way because it’s easy and expedient? OR do we go all-in to innovate some other solutions?”If it’s the latter, it’s factorially harder. Bleeding edge stuff rather than low-hanging fruits.It necessarily requires a lot of activity in the cranium and in the code+making hands. It means inventing and doing something no one else has been able to do before => proprietary IP and USP.It’s not even about a plane’s paint job and its fuel load which are two separate problems.It’s about existing surveys being like bicycles when what we really need and want are helicopters.And the helicopter hasn’t been invented or built yet — so we have to do that, figuring out the paint and fuel load as we go along.Cranium take-off, indeed, :*).
Some problems are not solvable.Some problems are theoretically vexing but have little practical impact.If you are looking for someone who likes to solve problems in the pursuit of having a tangible impact on the world around them, @JLM:disqus would be your guy.That’s all he is saying. I lean his way, but you also strike me as one of the happier people I know. So, as you were is my answer.Good luck with the Disney opportunity.
:*), thanks. I’m an irrational optimist whose nature is to go ahead and solve those seemingly “impossibles” —@JLM, yourself and many in this community are amazing in the generosity and diversity of tips you share with us. It does help to get the “We’ve been through the fires, got burnt and this is a better way to do it” perspectives.My real-life friends try ALL the time to get me to build things that have been done before and that they know I have the practical skills to deliver. One of them last week said I should do an updated version of ‘Timeout’. A big mediaco offered me a well-paid job in their Data team after a meeting. Someone else wanted me to do a version of ‘ChubbyBrain’ for Europe.But, you know, I could work with a team and ship those ideas within 6 months and be bored to tears.Instead, I apply head+hands+heart to Startup #1 (the root-level invention which is really really hard to do) and Startup #2 (with my Chief Scientist, CTO and Data Scientist making cool IoT fun that hasn’t been done in our way before).So … as I am, :*).
Fred gave a recommendation for a quick/easy/inexpensive survey tool… and you went off on a rant about ratings systems. I don’t disagree with you but one is not the other.
Thanks, Stephen. Critical reasoning and challenging each other’s thesis with source links is one of the strengths of AVC community.Fred would HATE it if we were all “Yes, Fred. Three bags full, Fred. Everything you like is amazing and we agree with you, Fred.”We all respect Fred in our own ways for different reasons. I certainly hold Fred in the highest regards.Still, if TechCrunch has an article which highlights the flaws of surveying for opinions with 5-star ratings and the first thing I see when I visit the survata site is a 5-star poll then I’m going to provide that TC link as well as MS’s negative experiences from years of using surveying systems of the same ilk to measure and model its employees by.
Agree with we don’t have to agree with Fred.Agree with stack ranking is the most evil destructive thing a personnel department has ever thought up, any personnel department that wants or agrees to do it should be disbanded. I personally told an HR manager that wanted to impose it on our team after it was acquired, why it is so bad (like your picture) and when he tried to go around me told him to crumple it up and shove it where the sun doesn’t shine.Disagree on that this post had anything to do with Stack Ranking, which not only fucks up peoples lives, but causes them to try and fuck up other people’s lives.Disagree that rankings don’t have some value. When I see a restaurants or Amazon review, I can directionally get a feel for the product or service. Yes, there is going to be that one “crazy” that has some rant, but by looking at the number and the general direction it provides a ton of value.
You’re right, Phil.Rankings, probability and surveys all have some value: bits of quantitative value.We can indeed infer some information from the number and the direction (up or down).The question is how do we gain more of the QUALITATIVE value and do those cardinal (8 votes up) and the nominal numbers (1st, 2nd, 3rd) tell us, for example, that the restaurant is “edgy”, “French”, “delicious”?
UK Election 2015: UK Polling Council launches independent inquiry into “apparent bias” in polling/surveying methodology:* http://www.thetimes.co.uk/t…TV segment on Sky News reported they’re going to examine the structural issues with surveys and polling methodologies which have been with us since the 1930s.* http://news.sky.com/story/1…Hopefully, some better tools will emerge for measuring decision biases for all of us.In the meantime, existing surveys can just “Keep calm and carry on as normal.”@disqus_Awy3Cl8ObF:disqus — well, we know the methodology is “square peg, round hole” and I specifically believe Probability isn’t the en pointe tool to measure human bias with (which is why I exercised my brain cells to invent a better system).Very very hard to do but already done.@JLM:disqus @pointsnfigures:disqus @jameshrh:disqus @falicon:disqus @SubstrateUndertow:disqus @stuartkmarvin:disqus @philipsugar:disqus @hymanroth:disqus @disqus_iQvLWp5GhS:disqus
Thanks for joining the conversation – I work at Survata. You aren’t wrong about the limitations of 5 star reviews, which is why it’s only one of 12 question types that we offer. We also don’t power performance reviews, for many of the reasons you mention – surveys simply aren’t the best tool for them. Most of our clients choose to run multiple surveys on the same subject, using multiple question types to learn as much as they can about their survey subject.
Alec, my use case needs in surveying systems are very specific and granular whereas most surveying systems (including survata) provide the standard formats which people are familiar with because those question types have been around since the 1920s and 1930s with the creation of Marketing and Advertising as social science disciplines.Now, even if survata doesn’t think it’s using 5-star reviews in the multiple question types, it probably is.If there’s any question that involves, “How much do you agree or disagree with this statement…Strongly disagree…Slightly disagree…Neither…Slightly agree…Strongly agree.”That is a 5-star scale right there. If there is text-mining by applying Machine Learning on the free-form input text, there is 5-star scale in the Positive-Negative Affect Schedule to decide if that comment is negative or positive.It’s not survata’s fault. It’s endemic in the way surveys have been designed since 1920s and 30s.Along came the Web and mobile and market researchers just went, “Well let’s take those survey designs from 1920s and 1930s and just, wholesale, digitize those survey designs. Flaws and all.Regardless of my outlier bleeding-edge views about survey systems and the mathematics behind it, survata will be just fine.People are used to surveying systems like survata and how they work. They use it. It does enough for what they need. They don’t need to think about how the data collection, analysis and recommendations generated by the mathematical algorithms affects the Bigger Picture involving some of the hardest problems in Human+Machine Intelligence.So I’m an outlier use case whereas Survata is for the same audience that’s familiar with Google Consumer Surveys, SurveyMonkey, Facebook surveys et al.
Ask Stoppleman what he thinks.
Thanks, Mark. I will ask him when I see him. Werner Vogels, CTO of Amazon, has given me his views on 5-star ratings and reviews.Yelp has gotten its share of criticism about its surveying methodology over the years. Regardless, 10+ years of data and users is still valuable to someone as a potential acquisition.It boils down to what a brand / user wants to survey for. Developers know the system’s broken but it’s used because it’s been around for 80-ish years (since 1932-ish).It exercises the minds of product people and developers from Yelp, Netflix, eBay, Amazon, Google etcetcetc:* http://www.quora.com/Is-the…The surveying, 5-star and reviews method has been a standard in people’s recent memories.However, it’s not necessarily the only or perfect method.Over the next decade — particularly because of Machine Intelligence — we’re going to be digging a lot deeper into standards that were and standards that SHOULD be.
for the record – i’m with you – just don’t want to discount that the very thing you criticize is a 10 Billion dollar “thing”
I appreciate you pointing it out because it’s true; the market values Yelp highly so it must have utility.Now………The global GBP in 2014 was approx $71 TRILLION.A large proportion of companies are graded by S&P / Fitch etc ratings as well as reliant on 5-unit scale weightings (aka 5-stars).So it’s not just a $10 billion market the alternative system to 5-stars would be useful and could become a utility / standard for.It’s the entire market of Data and how systems apply that data — in finance, in commerce, in industry, in machine learning and more.Of course Yelp’s $3.6 billion valuation (based on Yahoo! stock info today) is fantastic for the company.Still…there are systems and tools being made which aren’t yet standards or utilities like existing surveys and 5-star systems that may prove to be more valuable.Creating a whole new paradigm for data, even.
Curious if you may share what types of survey topics you have used it for, and was it to confirm something or have you gained solid new insights that were bankable?
I’m not going to share that publicly. Too confidential
😉
Brings up the issue of whether when you do these surveys the survey company should even know that you (USV) is the buyer vs. a random straw entity.
Typically, there are thousands of different surveys running on every topic you could imagine. Confidentiality and data ownership are key in these relationships. Clients has diverse as PE to CPG to Entertainment all participate.
And how effectively is survey fatigue managed?
Our exchange sees ~300-400K survey takers a day, so most user aren’t fatigued through multiple survey attempts.
Great question! We actually frequency cap respondents to address this issue.
Great point! We agree, confidentiality and ownership of data is critical to our space. That’s why our clients own the responses that they purchase, and we’ll never share any of their information without explicit permission.
@wmoug:disqus as the backbone of many survey platforms, we see thousands of survey fielded everyday across a wide range of topics: PE, CPG, Analysts, Entertainment… confidentiality and data ownership are key.
Fred; This blog should an elective class in MBA somewhere.The discussion and topics are fantastic.I would seriuosly consider sharing it in some form with business students in some school.I enjoy participating.Great forum.
I think that Fred actually did some mini MBA courses a few years ago. I am sure he has toyed with the idea of launching a course / curriculum in VC in some virtual school.
We’ve done thatIts called MBA MondaysYou can find it herehttp://avc.com/archive/#mba…
You can audit Illinois MBA on Coursera for free now. If you want a degree you can do it online to but then it costs money
Wonder if there is a way to create some kind of survey system for every time you are in the airport. You can fill out surveys instead of paying for Wifi. Same for planes.
Google offers this for site owners to let users view content. Slightly annoying, but I do prefer it over viewing ads or being told that I need to pay to get past the paywall.
Thanks for sharing 🙂
Here are the audience panel + survey platforms I aware of:1) Google Consumer Surveys2) Survey Money + Zoomerang3) Survata4) Instant.ly5) AYTM
We love Google Consumer Surveys. Quality is lower, but 10 cents an answer within 24 hours?? That’s tough to beat if you’re debating which image to use on your landing page..
So good re Jetshttp://www.newyorker.com/hu…
You either laugh or cry
The new GM, Maccagnan, seems to be the real deal. I can only wonder what Rex is thinking when he sees the cupboard being legit restocked.
Same with the Bears. But at least we aren’t the Jets
Jets have cornered the market for bad
I wonder if something like Palantir (Palantir.com) could be constructed for investment purposes?
The value prop of omnibus or panel study research is best served when the evaluative criteria you set is reasonable. Too often it isn’t. The data should be viewed directionally as “off the shelf” survey tools, like Survata, frequently have inherent shortcomings and biases, for example how they initially aggregate and screen respondents, which can be further exacerbated when companies seeking the learning are inexperienced in questionnaire/survey design. That said, panel surveys in my experience can and do provide limited value (e.g., initial concept testing).
Thanks for the comment. We source our respondents through our publisher network, who gate content and experiences with our SurveyWalls (a maximum of 6 questions with up to 2 screeners). This reduces response bias that can be inherent in panels (we do use panels when the targeting requested is highly specific and would cause the response time to be prohibitively long through our Publisher Network. Panels are only an option in our Advanced Survey package).Regarding your comment on inexperienced survey design, all surveys launched from Survata are reviewed by a survey analyst who has advanced degrees and extensive training in both market research and survey design, something unique to Survata (as far as I know). Clients often ask us for help writing questions or designing surveys, so we put a lot of resources into helping them.Let me know if I can answer any other questions!
Thank you. I have a heavy market research background and have used Google surveys, Survey Monkey and my personal favorite to date, AYTM. I will try Survata. Appreciated.
Great to hear! We’re looking forward to hearing from you and helping out!
Dear FredI am slightly disappointed. :)Here is why:We are in 2015, the web has no secrets for you an you say- $1-$2 per complete is not a lot- Three to seven days is what you consider “quick”- Survata’s UX is simple, easy, fast, etc…My questions to you are :1. Did you ever look for better solutions? After all you invest in market places and Market Research is a great industry which is still looking for a leading online market place.2. Do you think there is room for a leading online opinion market place? Or maybe there is one already ?3. Do you think that Survata’s YC label is what made you go for it in the first place?Some people already mentioned alternatives in the comments but here are a few based on my own comments. – VoicePolls Reports offer 10,000 opinions for $500, not just 500- Toluna, AYTM, VoicePolls, and others do in a few hours- TypeForm’s UX is simple, easy, fast- Etc..
Is Survata a YC funded company? I had no idea. My colleague selected it and we use it and we like it. Enough said
Yes, we are YC-backed (S12). We’re always happy to get referrals from our YC-associations, as we are strong believers in YC’s strength in identifying great ideas and companies, but are just as excited to see customers who have no idea about our YC experience enjoying the product!
@melkisch there is a marketplace for survey sampling… Fulcrum and we run it here at Federated Sample. Survata has been a client and so has the rest of their supply chain. Part of the reason survey responses have gotten faster, better targeted, and less expensive is because of market place dyanmics. Many of the companies listed here are our clients.
Do you think that Survata’s YC label is what made you go for it in the first place?You say that as if being a YC company is like the “Good Housekeeping” seal of approval.
Great questions. I work at Survata, so I admit to some bias, but to give you some information from the source:1 – Nearly all of our clients have been using other solutions for years before discovering Survata. In our customer interviews, the most consistent feedback we receive is that we are able to deliver higher quality respondents more quickly and for a lower cost. Additionally, many of our clients continue to use other market research and survey tools as a supplement to Survata. Because we source respondents in a somewhat unique manner, comparing our results with those collected via other methods can often be illuminating, especially given that we are generally substantially cheaper than other tools.Regarding the competitors you mention – in our basic package, respondents are never paid per response, which improves response bias.
As USV isn’t only focused on US, On Device, funded by Passion Capital in London offers surveys by mobile phone on emerging markets consumers – https://ondeviceresearch.com/
We have been paying between $1 and $2 per complete which, for us, is not a lot to get some answers quickly.Well if you think it has provided valuable information then fine. But I want to point out that I’ve seen these surveys before. They have typically popped up on paywalled news sites as a way to be able to view a story that previously you might have been blocked from seeing.So in other words you will click on a story link and then all the sudden there will be a survey “Do you drive a motorcycle”. Can’t vouch for others behavior, but I typically answer the questions as quickly as I can with nominal regard to my answers just so I can get to where I am going.In fact Survata says:We find respondents to complete your survey. For our Basic Surveys, we partner with high-quality online publishers (Are you a publisher? Learn more.), whose visitors take surveys to access content like an e-book or video (see a demo). For our Advanced Surveys, we partner with research panels that have pre-recruited respondents to complete surveys. It’s definitely not only an e-book or video that is for sure. So that entire targeting idea is suspect just on that.I had no idea they were charging $1.00 (in US) for that info I wonder what percentage of that they give to the news site. Great business idea though, hard to believe that they will not face serious competition on what they are charging for this service.
Thanks for the comments, we appreciate all the feedback we get. Apologies for being unclear – we never meant to imply that our surveys would only appear on ebook or video sites, just that those are two of the common use cases. It’s up to the publishers in our Publisher Network to determine what content they want to gate (if you are a publisher and are interested in joining our network, please shoot a note to publishers AT survata DOT com).Also, in the example you give, where you mention you respond as quickly as possible, the client would never be charged. We only charge clients per validated response, which we clean with a combination of software and human review (and, as we provide the raw responses to clients, if, for whatever reason, the client is dissatisfied with a response, we’re happy to review it again, and will refund the client for any mistakenly validated responses).
Survata CEO here. Thanks for the mention, Fred. Happy to answer any reader questions!
need it all to be programmatic – i am now waiting for a “consultant” to call me…..i need answers quicker than that
Hi Mark,Thanks for trying us out, and for your feedback. There are still some human touch points in our service; yet we’re of course working to automate anything that can (and should) be automated.
I think you have to go back to the basics and remember the critical elements of marketing research and not focus on the data collection tools because there is no doubt that Survata is exceptional (as are many other online survey tools) and Fred is dead on correct to use this tool to get great directional data to give him and his team a good gut feel on whatever he needs to get a gut feel on. What strikes me the most about this: today’s survey tools are perfectly in line with the lean start up and getting out of the building concepts we now know are critical to start up success – we are so lucky today – get knowledge now and inexpensively – Survata is one of the good tools to do that. Fred knows. I am all for it.The basics: questionnaire design is critical (e.g. no double barrelled questions, proper length etc.) and obtaining the opinions and attitudes from a representative sample has always been the fundamental challenge and requirement – even in the old days of using the telephone as a data collection tool. All of today’s online survey tools liberated the masses and allowed individuals to take control and be their own marketing researcher. Not always a great thing in some people’s hands, but it changed an entire industry – in effect dramatically displacing a lot of professional marketing researchers and putting many larger traditional research houses on the defensive (or out of business). And for the better overall – conducting research in the old days was hugely expensive and took way too much time – resulted in many poor product and business decisions being made (too many examples to list). It also allowed many professional marketing researchers to become entrepreneurial and open their own research shops – which is great. I love the technology today and if you apply some marketing research survey design basics and best practices (which can be easily learned), a start up or large corporation (or a VC!) can get great directional data quickly (like in hours if needed) and at a fraction of the cost of hiring a large firm (not that hiring a large firm is a bad thing) – just depends on your needs. Re: sample – obtaining sample via telephone was really always not that representative or reliable (an old industry dirty secret), was just that a larger sample compensated for the downside. Sample quality has and always will be a factor – online sample be it from panels or randomly selected from sites can actually be far more representative of the target audience you are trying to gain opinions and attitudes from than ever before – not perfect, but just as good as it has always been (there have been professional survey takers for 75 years). Keep to your research objectives, use research rigor to create a great questionnaire, get opinions and attitudes faster than ever before and at a budget that helps to reduce the burn rate by using Survata yourself (or hire a great marketing research consultant who will use Survata) – you are on your way to getting knowledge to help create the product that is going to solve that screaming problem for your customers. It is good to be alive today!
Survata team member here – thanks for the shout out @fredwilson:disqus. We’re happy you’ve found us useful at USV! Happy to answer any questions.
Survey Monkey was doing $113m in revenue and $61m in ebitda in ’12. I look at a lot of companies and rarely, if ever, do i see a company that profitable at $113m in revenue. A simple product with a clear value prop can only generate that type of profit margins. On the flip side, it just begs for competition. I just wonder what their ebitda margins look like today with the ever increasing competition.
I’ll add that the reason I really like survata is their supportEvery survey gets assigned an analyst that helps you fine tune the survey in order to get better resultsThey’re very responsive and very helpful
Thanks! We definitely pride ourselves on being as responsive and helpful to our customers as possible, it’s always great to hear when someone has a good experience!
I found it interesting that you on your end would pay $1 to $2 per answer on a survey to get answers quickly in the customized way you need. So let’s say that might mean $1000 because often that’s the size of a geographically-weighted sample. And that’s how a survey company makes money and that’s fine.My problem is the tiny percentage in mils we survey-fillers get out of this. I joined several of these companies that offer to pay you and I come to find out what a scam it is. First, if you are poor, old, or work in the media or all three (a terrible demographic for sellers), they disqualify you for many surveys, but often don’t tell you that until you’ve completed the form – and lost the fee. Then while it looks on the panel as if you have “$30” compiled, in reality, those are only 30 survey-world virtual dollars that translate only into a $5 coupon at Olive Garden — which you could get just by watching your local shoppers.The exception is Knowledge Panel. First, they give $10 to join. Then, they don’t seem to pay you anything after that, but they have really interesting surveys that aren’t the usual inane survey company formula to ask the same question 3 different ways to try to shake you loose from an answer. They also have scientists who give their names and emails and you can write comments to them and have conversations. Then many times I would find the survey terribly rigged, especially with that technique of triadic repetition to wear you down to get the answer their client needs.Then there’s the fact that surveys are surveys of people who take surveys. What is it, 2% of Americans? Indeed, I began assiduously to get on political surveys because I was tired of my opinion on politics in NYC not being represented.
Sorry to hear that you’ve had a bad experience as a paid survey respondent. Just to clarify – we do not pay respondents to our Basic Survey package. Respondents complete surveys in order to unlock content or experiences on our partner websites.
I am curious what you used for. Can you be specific? 🙂
‘focus group planners’–is that actually a career?i know that the movie biz still uses live showing to market test cuts of a film but beyond that this is just so MadMen honestly.
Great question – our clients are only ever charged for validated respondents. Before delivering your final results, we clean all responses using a combination of software and human review. Additionally, you’ll receive the raw responses that you are charged for, and if you have any issue with their validity, we’re always happy to take another look and discuss.
That’s given us sequel after sequel. Yawn
Alec you make two great points. The first is opinions as a form of currency — anything that has value and is fungible is currency. Putting together Arnold’s assertion that organizations crave user feedback and your assertion that these opinions can be traded, will, in my view, create a large new market going forward.The second regards bias. There are survey companies I could mention who claim they can get people with household income above $150K to spend 20 minutes on a survey for $2. I don’t believe them, for pretty obvious reasons.The utility of $2 varies greatly with how much you are worth, and I’m actually not too convinced that even people on lower incomes would sell their time at the equivalent of $6/hour. I suspect most of the people on these panels are teenagers with fake profiles.The paywall model on the other hand offers constant utility. The desire to access the content is not influenced by net worth. This means the demographic profiles of your users are much less likely to be fake.Good luck with this.
Thanks AlecAppreciate this clarification and wish you all the luck with this.I have strong biases on this I admit and they are not about your company but sit at the intersection of the value of opinions as currency and in my mind the difficulty in using surveys as a tool to plump and aggregate them.Thanks for jumping in with this.
Thanks David. While we do offer panel fulfillment in our Advanced Package, we limit this to surveys where the targeting is highly specific, requires some amount of vetting, and is time sensitive, as you correctly identify a potential weakness with panels. Again, we review all responses, and only charge clients for those we’ve validated. We also only work with panels who consistently deliver high quality results. To your point asking why high paid, wealthy people would participate in panels, as I understand it, they generally are not paid by response, but for their time. The panels also often compensate those they can validate in other ways, such as airline miles that they award to the company employing the respondent, which helps ensure validity (although it is still far from perfect). This is why B2B respondents are often substantially more expensive, as the verification process takes more time, and is substantially more expensive.
No problem! You’ve correctly identified some issues that are somewhat endemic to market research. We’re always working on addressing what we see as problems in the space. It was actually our CEO’s time as a consultant and analyst at a VC firm that supplied the inspiration for Survata, as he encountered first-hand the problems common to market research surveys.