Feature Friday: A/B Testing Headlines On WordPress
AVC community member Shana Carp has been building a neat service that A/B tests headlines for WordPress posts and helps you figure out the one that will bring the most traffic.
It’s called BayesianWitch. You can sign up here and add the plugin to your WordPress.
Although I think it’s a great idea for someone who wants to optimize their posts for more traffic, I have not implemented it here at AVC.
As I told Shana, I like to write my own headlines and I am not that concerned with traffic. There’s already a healthy number of people who come here every day and engage.
But if you are still building your readership and want to make sure you’ve got a headline that pops in social media channels like Facebook and Twitter, you should give BayesianWitch a try.
I am sure Shana will be hanging out in the comments and please let her know what you think about the service, how it works, etc, etc.
Comments (Archived):
Shana: I just wanted to say I’ve been following the posts on Medium, it’s interesting stuff, good luck with it.
Go, Shana!
Sounds like a great plugin… worthy of pluggin’Good luck Shana!
the punster never sleeps 🙂
Shana is not going to like this but when I tried to download the plugin, I got a 404 message of file not found.I knew she had been working on this and I had intro’d her to some potential customers. Glad to see it’s off the ground and running.
You can install it straight from the WordPress Admin- Add New plugin and it’s there.
<redacted> found it and installed it
Bayesianwitch (in one word)
got it, but now can’t install because I can’t connect to the host.
to your host or us?
Can you give me the exact url -because this one works http://static.bayesianwitch…Edit: we think it was cloudfare…
Weird. I logged in and now it shows it installed!
Why do you even need cloudflare for your site? You don’t need any stinking preservatives (raw if fine) you’re not serving up a bunch of static content. How big is the plugin download anyway? You can just put the plugin somewhere else alone if that were a problem.
Congrats Shanna.
Off-topic, but I’d love to see some response to the disastrous/totalitarian/innovation-stifling proposal for the BitLicenses (i.e. licenses for cryptocurrency startups) in New York: http://www.reddit.com/r/Bit…If this goes through, it is pretty much guaranteed to completely kill Bitcoin startups in New York.
That’s an opening proposal. It is subject to comment and discussion
True, but if you start from a completely messed up place its hard to end up somewhere good.
Big Bitcoin convention here in Chicago this weekend. I am going.
Suppose you have more things that can be changed than just the headline and want to know what changes to make? How to do that? Try all possible combinations?Enter the subject of experimental design or analysis of variance. Good source? Sure: Who needed to do a lot of that? I mean, here’s a field: Now what seed, fertilizer, herbicide, cultivation technique, planting time, watering, etc., will give the best yield? Yup, you guessed it, the ag department at Iowa State University. So, sure, here it is:George W. Snedecor and William G. Cochran, ‘Statistical Methods, Sixth Edition’, ISBN 0-8138-1560-6, The Iowa State University Press. Ames, Iowa, 1971.My math background gives me the prerequisites, and more, for that book, but, sadly, I’ve never gone through it. But my wife went through it very carefully and was good at it. Why? It’s one of the basic tools in experimental work in the social sciences.This would have been a very good time for her: She was essentially a mathematical sociologist, studied under James Coleman and Pete Rossi, both past Presidents of the American Sociological Association, and, thus, would be a great choice as a ‘theoretician’ of social media, Web site testing, etc. Yes, they had her studying graph theory, too!For the arithmetic, use SPSS, SAS, R, or some such? Or just write some code, say, starting with a subroutine library?
Congrats Shana
Congrats on releasing it into the wild, Shana!
Re: HeadlinesI don’t read many newspapers anymore…but I truly enjoy scanning the headlines from the Post and the Daily News.#citythatneversleeps
We recently ported our entire publishing site to WordPress, and it’s been a wonderland of plugins after being in a closed CMS. This will be a great addition I think.
test, and tell me. It should help keep people reading (whole point of title testing is to keep people reading) though we do weigh in some other stuff
Anything called “Bayesian Witch” deserves to be immediately downloaded irrespective of functionality or purpose.
I should tell you the name came from wondering how to combine bayesian statistics with the movie Bedknobs and Broomsticks.Also the logo tweets at you and sends emails…
“You WordPress. People will read with our app magic.”?We give you the same tools the big publishers have to keep readers engaged.<3 Angela Lansbury and that movie, especially #Bobbing along, singing a song in the beautiful, briny sea#.You’re essentially applying Bayes to help people spot the diamonds in the blog sands better. It’s not obvious at point-of-entry on that page that it’s for WordPress and an app/plugin is involved.
https://www.youtube.com/wat…And yup.
Congrats Shana. Build, learn, iterate – step one, checkkkkkk
Go for it @ShanaC:disqus
Skiing off-piste. I never liked the term ‘plugin’. It immediately diminishes perceived value – ‘it’s just a simple plugin’.
well technically speaking its a wrapper for an API. But we figure with about 18% of the cms unvierse on wordpress, it makes sense to just pre-wrap, Most people are not going to implement an API. *shrug*
I just think of them as apps for websites
exactly
Totally get that…especially as WordPress brand their “App Store” – “WordPress Plugin Directory” FFS. Wonder how history would have embraced the ‘iPhone plugin directory’I’m sure there’s a ton of work and services behind your ‘app’. It just seems diminished to term it a plugin.
At my work when I started trying to use an API for one of my vendor products they were concerned I was “hacking” it and I had to explain in a meeting that when vendors expose APIs they usually WANT you to use them.Suffice to say I think there is a large population that does not understand APIs but uses WordPress and might have interest in A/B testing, especially if they see and use it in their e-mail program.
.+10 for “off-piste”Well played.JLM.
Picking the best title is an old technique that Huffington Post mastered a few years ago. Actually, that was a key growth hacking tactic they employed. They would try 2-3 different titles to the same article and monitor traffic for the first 30 minutes, then switch to the one that got the most traffic.I’m downloading Bayesianwitch and trying. Thanks Shana.
we automatically make the switch for you. My cofounder is one of the leaders in the US of the kinds of algorithms huffpost uses.
Great. 1/ How quickly after the post do you make that switch? 2/ How do you handle the URL syntax for that post? does it get changed throughout, i.e in my WordPress and for Google?
1) Depends on site. You need roughly 200ish views per arms for optimization to start happening2) We hash urls, the first title on wordpress is usually the url, and we therefore don’t have any Google issues
1/ what is “per arms” mean? 2/ so the initial permalink url doesn’t change even if the title changes? i don’t understand that part of the magic. because my urls include the title.
1)per title.2) so lets pretend your blogpost is wordpress.net/hello.htmlyour titles are – hello and hithe url for hello would bewordpress.net/hello.html#th…and the url for hi would bewordpress.net/hello.html#gh…the first tile is part of the url – but the title is determined by the hash. Google ignores the hashes.
Got it. So if I share the first tittle’s URL, it should be ok?Do you advocate waiting til the best title is found before sharing on social media?
You’ll be fineAnd no – getting people inbound to your site is different than what happens once they are there. For many people/sites you want them to read more, even if they came inbound by one technique So share early and often 🙂
The idea @ShanaC:disqus of title as the key for social pickup on predominantly graphic nets is non intuitive to me.I like this plug in.Extant though from graphics, is it really the criteria for choice for the social nets unless you are ABing within a single graphic context?
a lot of behind the scenes audience development is extremely not intuitive.
My cofounder is one of the leaders in the US of the kinds of algorithms huffpost usesWhich cofounder? If that is the case should appear on the “about page”.
Chris. Jordana is not a real person. She just appears in marketing messages.
Shana, congratulations on pushing the baby out! Sorry the witch…l love the name.We’re not on WP, or I’d try it out.
we have an api. (docs are here http://www.bayesianwitch.com/docs/... ) its a wrapper for the api technically speaking. You could implement it if you want.
Just curious- how about offering a “predictive” best title i.e. give me some data on various titles before I submit the post, not after. Is that possible?
Good idea. But decided on what basis?
we’re working on ranking your titles next. Predictive is very very hard.
I don’t use WordPress but am a big Shana fan and LOVE that you used today to give back to the AVC community a bit (though tech. you give back to the community everyday with your posts/attention).
i try hard?
You *always* give more than you get…I’m always a HUGE sucker for that 🙂
oh – we have an api. I totally forgot that when reading thisIt is here: http://www.bayesianwitch.com/docs/...
This post does a much better job of describing the product than the homepage for it. Hint, hint, Shana. :-)”A/B test your headlines. Download the WordPress Plugin Now.”
we tested with publishers- a lot of people don’t know what AB testing is.Also, strickly speaking, you’re not AB testing. You’re Bandit testing. Opens up a whole can of worms
You’re testing your copy, so I’m sure you’re optimizing. Still, I’d think about something that clearly explains what it is I’m downloading.In the part up top where it says “Give yourself the same tools…” I’d think about something like “Our WordPress plugin allows you to test multiple headlines and know which one will get you the most readers.” With a button that says “Download the Plugin”Not saying that’s exactly it, but something that clearly states what the customer is getting. And that’s the end of my two cents. 🙂
Agree. Bullet points might work well with stating multiple benefits.
Shana – I’m not a fan at all of any site that tries to hide some basic pricing reference. I think you need price points beyond “hobbyist = free” and “deep pockets = call us”. Which appears only if you click the sign up at the bottom and not at the top (assume that’s a link that needs to be fixed?).There has to be a reference price wise of what people are getting involved in. And not only that but the price reference should be right on the home page. There should be a pricing link right at the top.You could always shorten the name to baywitch.com by the way. Or witchian.com or even better, and sure to get attention:bitchian.com !(Btw you need to get baysian as a typo of bayesian …)
we know this. its related to incorporation problems. The state of delaware is a fun place to call…I assure you 😉
Not clear if you are talking about my comments relative to the name or with regards to the pricing? Either way I don’t see how incorporation issues could be involved in either. Separately, you can be XYZ corp and trade under “Shul Labs”. (Which is open by the way…)Jordana’s name, bio, and icon, by the way wasn’t far off enough to scream “actually is the office dog” which is why I thought it was a real person. I actually googled it.
Relative to pricing.The long term goal is to get the mascot to be the customer success person. We’ll see how that goes
Bandit testing is far, far superior to A/B testing, a conversion tool favored by many publishers today. Since Bandit is a bit more complex to explain than A/B testing, it might be beneficial to include a couple of use cases (and testimonials) off of your home page. Bandit can be both more effective and efficient than A/B testing.Wish you much success w/ this venture…Certainly addresses a legit need.
We radially shrunk the product for basically this reason. Keeping it to headlines 🙂
I hear you, but are your headlines perhaps a bit too vague? (I’m kind of in Brandon’s camp here.)Every publisher understands and likely has experimented w/ A/B testing, while Bandit testing delivers a higher level of optimization.Why not position your company comparatively to A/B testing? A/B testing is like The Flintstones, while Bandit testing is like The Jetsons.”Still relying on A/B testing?… Upgrade your optimization tools w/ BW to enhance readership.”…..Just saying. Love the name, btw.
Small publishers tend to be weaker in that regard. if you are big, then they might – but you’d be surprised how many large publishers don’t understand the idea of a two tailed test, for example.
Hey ShanaTwo requests:1. How does this work to the extent you want to share? I this SEO based at its core?2. Many bloggers should be interested but their traffic sample is low–maybe 1-5K per post. How large a sample is needed?
1) No, it is not SEO based at its core. You may use SEO to get people into your blog, but now that they are there, how do you get them not to push the back button and read more articles? You show them more potential cotnet – usually a title, and in some cases images as well. But what is the right title to make them stay and read another article? And what is the right title to make them share an article? (that’s the question we’re asking)2) The minimum for testing two titles is 350-400 pageviews. While the test won’t “finish”* you’ll see uplift nonetheless. So our minimums are way smaller than the bloggers you are talking about. And it is designed that way on purpose.*Its not designed to really finish, it will react to traffic as it continues to happen.
I like!If you are selling this to the long tail of bloggers out there you have some interesting work to make it really simple to understand.Looking forward to your blog posts!
what would you suggest as a starting point?
“what would you suggest as a starting point?”I think you need to get a bunch of more or less clueless people [1] sprinkled in with some more knowledgeable people to review your marketing content and what you are offering. You need both groups to understand what you are selling. You aren’t either person you know to much already.The headline might be: “Why stupid people can help you with yourmarketing and smart people can’t”[1] Of different levels of intelligence.
we’re trying that out actually. Its why we nixed the data science taglines. Most people felt overwhelmed by data science, or that they would lose control.
Not a believer in asking your market what to do.Where to start? The beginning.Your sense of the value of the product to a segment of the market is where to start.And wherever that is–be it me or someone more or less technical, make it clear, make the choice easy.
Arnold: “How does this work to the extent”I think you need to put more on the site to this end. I think some people know nothing and just trust in a marketing message. Others who know more won’t believe in something unless they have some clue as far as (in non marketing speak) what is going on under the hood. Otherwise they make assumptions (because they know something) and those assumptions might be wrong.Part of this is the curse of knowing to much. If you know little you tend to take things on faith sometimes. If you know more you tend to want to know how something is possible (because you have a frame of reference).Here’s an example. If you told some people “so with this app you can walk into Starbucks and they will have your drink ready” they might just think “geez you can do anything with an app that’s great!!!”. But if you told me that I’d want to know how that was being done and how it was even possible. And I might actually miss a few great features of the product because I might assume that it might implement a way of providing that benefit that I wouldn’t approve of.I don’t think you need to clutter and confuse the first group of people just make sure the info is available for the latter. Short blurbs on homepage and more detail in a faq.
Sorry, not following.
I was just attributing the quote to you.( I wasn’t asking you a question.)Or do you mean that you don’t understand the points that I am making?
none of it honestly, inlcluding the analogy between a starbucks card and a WP plugin.this is a very complex product.tactics are kinda useless till they figure out what it is and who will buy it.
Yeah, Shana! Hope it does really well.
Congrats, Shana. Hope you feel better soon!
i can tell you I do not recommend doing a 7+ mile hike with bronchitis…
.Congratulations, Shana. Good luck and crush it.JLM.
how about “Google Bitch”?the web is becoming ever more tabloid.
I’m currently using Noah Kagan’s KingSumo headlines: https://kingsumo.com/apps/h… It’s the same idea and I like it very much 🙂
its not – we’re tracking other things for a better optimization experience. My cofounder is also probably the lead person in the US for implementing these kinds of optimizations.
Rename it ‘Trollbitch’ ??
i’m in colorado recovering from bronchitis/really bad chest cold.
Never know what you might learn :)Learning is not always good. Sometimes it can mess with your gut. I never have read any books on negotiation and never plan to. I don’t want my brain messed up with other people’s methods or what works for them. I have something that works for me that was developed over time. Not interested in something that will derail my gut.My guess is that Fred likes AVC “just the way you are”. More visitors might mess with the delicate balance between regulars and drive by visitors who could ruin the atmosphere. Then next thing you know everything changes. Not to mention that he’s probably tapped out with the amount of commentary there is now. He has a day job. It’s not a hobby.I would think that the product is good for larger sites that have to churn out many headlines and make money primarily by advertising as opposed Fred’s use as business development and to talk about what is on his mind each day.Personally I would think it’s fairly easy to reverse engineer what makes a good linkbait headline. I mean if you keep your eyes open and pay attention (to what others are doing) it’s kind of obvious.Fred could have just said “Why I don’t do A/B testing” if he wanted to get more people interested in the post for example.