Feature Friday: A/B Testing Headlines On WordPress

AVC community member Shana Carp has been building a neat service that A/B tests headlines for WordPress posts and helps you figure out the one that will bring the most traffic.

It’s called BayesianWitch. You can sign up here and add the plugin to your WordPress.

Although I think it’s a great idea for someone who wants to optimize their posts for more traffic, I have not implemented it here at AVC.

As I told Shana, I like to write my own headlines and I am not that concerned with traffic. There’s already a healthy number of people who come here every day and engage.

But if you are still building your readership and want to make sure you’ve got a headline that pops in social media channels like Facebook and Twitter, you should give BayesianWitch a try.

I am sure Shana will be hanging out in the comments and please let her know what you think about the service, how it works, etc, etc.


Comments (Archived):

  1. John Best

    Shana: I just wanted to say I’ve been following the posts on Medium, it’s interesting stuff, good luck with it.

  2. Anne Libby

    Go, Shana!

  3. JimHirshfield

    Sounds like a great plugin… worthy of pluggin’Good luck Shana!

    1. Alex Wolf

      the punster never sleeps πŸ™‚

  4. pointsnfigures

    Shana is not going to like this but when I tried to download the plugin, I got a 404 message of file not found.I knew she had been working on this and I had intro’d her to some potential customers. Glad to see it’s off the ground and running.

    1. William Mougayar

      You can install it straight from the WordPress Admin- Add New plugin and it’s there.

      1. pointsnfigures

        <redacted> found it and installed it

        1. William Mougayar

          Bayesianwitch (in one word)

          1. pointsnfigures

            got it, but now can’t install because I can’t connect to the host.

          2. ShanaC

            to your host or us?

    2. ShanaC

      Can you give me the exact url -because this one works http://static.bayesianwitch…Edit: we think it was cloudfare…

      1. pointsnfigures

        Weird. I logged in and now it shows it installed!

      2. LE

        Why do you even need cloudflare for your site? You don’t need any stinking preservatives (raw if fine) you’re not serving up a bunch of static content. How big is the plugin download anyway? You can just put the plugin somewhere else alone if that were a problem.

  5. kirklove

    Congrats Shanna.

  6. Brian Crain

    Off-topic, but I’d love to see some response to the disastrous/totalitarian/innovation-stifling proposal for the BitLicenses (i.e. licenses for cryptocurrency startups) in New York: http://www.reddit.com/r/Bit…If this goes through, it is pretty much guaranteed to completely kill Bitcoin startups in New York.

    1. fredwilson

      That’s an opening proposal. It is subject to comment and discussion

      1. Brian Crain

        True, but if you start from a completely messed up place its hard to end up somewhere good.

        1. pointsnfigures

          Big Bitcoin convention here in Chicago this weekend. I am going.

  7. sigmaalgebra

    Suppose you have more things that can be changed than just the headline and want to know what changes to make? How to do that? Try all possible combinations?Enter the subject of experimental design or analysis of variance. Good source? Sure: Who needed to do a lot of that? I mean, here’s a field: Now what seed, fertilizer, herbicide, cultivation technique, planting time, watering, etc., will give the best yield? Yup, you guessed it, the ag department at Iowa State University. So, sure, here it is:George W. Snedecor and William G. Cochran, ‘Statistical Methods, Sixth Edition’, ISBN 0-8138-1560-6, The Iowa State University Press. Ames, Iowa, 1971.My math background gives me the prerequisites, and more, for that book, but, sadly, I’ve never gone through it. But my wife went through it very carefully and was good at it. Why? It’s one of the basic tools in experimental work in the social sciences.This would have been a very good time for her: She was essentially a mathematical sociologist, studied under James Coleman and Pete Rossi, both past Presidents of the American Sociological Association, and, thus, would be a great choice as a ‘theoretician’ of social media, Web site testing, etc. Yes, they had her studying graph theory, too!For the arithmetic, use SPSS, SAS, R, or some such? Or just write some code, say, starting with a subroutine library?

  8. awaldstein

    Congrats Shana

  9. Kirsten Lambertsen

    Congrats on releasing it into the wild, Shana!

  10. John Revay

    Re: HeadlinesI don’t read many newspapers anymore…but I truly enjoy scanning the headlines from the Post and the Daily News.#citythatneversleeps

  11. Jon Michael Miles

    We recently ported our entire publishing site to WordPress, and it’s been a wonderland of plugins after being in a closed CMS. This will be a great addition I think.

    1. ShanaC

      test, and tell me. It should help keep people reading (whole point of title testing is to keep people reading) though we do weigh in some other stuff

  12. Trevor Bodily

    Anything called “Bayesian Witch” deserves to be immediately downloaded irrespective of functionality or purpose.

    1. ShanaC

      I should tell you the name came from wondering how to combine bayesian statistics with the movie Bedknobs and Broomsticks.Also the logo tweets at you and sends emails…

      1. Twain Twain

        “You WordPress. People will read with our app magic.”?We give you the same tools the big publishers have to keep readers engaged.<3 Angela Lansbury and that movie, especially #Bobbing along, singing a song in the beautiful, briny sea#.You’re essentially applying Bayes to help people spot the diamonds in the blog sands better. It’s not obvious at point-of-entry on that page that it’s for WordPress and an app/plugin is involved.

  13. Russell

    Congrats Shana. Build, learn, iterate – step one, checkkkkkk

  14. Tom Labus

    Go for it @ShanaC:disqus

  15. Barry Nolan

    Skiing off-piste. I never liked the term β€˜plugin’. It immediately diminishes perceived value – β€˜it’s just a simple plugin’.

    1. ShanaC

      well technically speaking its a wrapper for an API. But we figure with about 18% of the cms unvierse on wordpress, it makes sense to just pre-wrap, Most people are not going to implement an API. *shrug*

      1. Jon Michael Miles

        I just think of them as apps for websites

        1. Barry Nolan


      2. Barry Nolan

        Totally get that…especially as WordPress brand their “App Store” – “WordPress Plugin Directory” FFS. Wonder how history would have embraced the ‘iPhone plugin directory’I’m sure there’s a ton of work and services behind your ‘app’. It just seems diminished to term it a plugin.

      3. Matt Zagaja

        At my work when I started trying to use an API for one of my vendor products they were concerned I was “hacking” it and I had to explain in a meeting that when vendors expose APIs they usually WANT you to use them.Suffice to say I think there is a large population that does not understand APIs but uses WordPress and might have interest in A/B testing, especially if they see and use it in their e-mail program.

    2. JLM

      .+10 for “off-piste”Well played.JLM.

  16. William Mougayar

    Picking the best title is an old technique that Huffington Post mastered a few years ago. Actually, that was a key growth hacking tactic they employed. They would try 2-3 different titles to the same article and monitor traffic for the first 30 minutes, then switch to the one that got the most traffic.I’m downloading Bayesianwitch and trying. Thanks Shana.

    1. ShanaC

      we automatically make the switch for you. My cofounder is one of the leaders in the US of the kinds of algorithms huffpost uses.

      1. William Mougayar

        Great. 1/ How quickly after the post do you make that switch? 2/ How do you handle the URL syntax for that post? does it get changed throughout, i.e in my WordPress and for Google?

        1. ShanaC

          1) Depends on site. You need roughly 200ish views per arms for optimization to start happening2) We hash urls, the first title on wordpress is usually the url, and we therefore don’t have any Google issues

          1. William Mougayar

            1/ what is “per arms” mean? 2/ so the initial permalink url doesn’t change even if the title changes? i don’t understand that part of the magic. because my urls include the title.

          2. ShanaC

            1)per title.2) so lets pretend your blogpost is wordpress.net/hello.htmlyour titles are – hello and hithe url for hello would bewordpress.net/hello.html#th…and the url for hi would bewordpress.net/hello.html#gh…the first tile is part of the url – but the title is determined by the hash. Google ignores the hashes.

          3. William Mougayar

            Got it. So if I share the first tittle’s URL, it should be ok?Do you advocate waiting til the best title is found before sharing on social media?

          4. ShanaC

            You’ll be fineAnd no – getting people inbound to your site is different than what happens once they are there. For many people/sites you want them to read more, even if they came inbound by one technique So share early and often πŸ™‚

          5. awaldstein

            The idea @ShanaC:disqus of title as the key for social pickup on predominantly graphic nets is non intuitive to me.I like this plug in.Extant though from graphics, is it really the criteria for choice for the social nets unless you are ABing within a single graphic context?

          6. ShanaC

            a lot of behind the scenes audience development is extremely not intuitive.

      2. LE

        My cofounder is one of the leaders in the US of the kinds of algorithms huffpost usesWhich cofounder? If that is the case should appear on the “about page”.

        1. ShanaC

          Chris. Jordana is not a real person. She just appears in marketing messages.

  17. Alex Wolf

    Shana, congratulations on pushing the baby out! Sorry the witch…l love the name.We’re not on WP, or I’d try it out.

  18. William Mougayar

    Just curious- how about offering a “predictive” best title i.e. give me some data on various titles before I submit the post, not after. Is that possible?

    1. Alex Wolf

      Good idea. But decided on what basis?

    2. ShanaC

      we’re working on ranking your titles next. Predictive is very very hard.

  19. falicon

    I don’t use WordPress but am a big Shana fan and LOVE that you used today to give back to the AVC community a bit (though tech. you give back to the community everyday with your posts/attention).

    1. ShanaC

      i try hard?

      1. falicon

        You *always* give more than you get…I’m always a HUGE sucker for that πŸ™‚

  20. Brandon Burns

    This post does a much better job of describing the product than the homepage for it. Hint, hint, Shana. :-)”A/B test your headlines. Download the WordPress Plugin Now.”

    1. ShanaC

      we tested with publishers- a lot of people don’t know what AB testing is.Also, strickly speaking, you’re not AB testing. You’re Bandit testing. Opens up a whole can of worms

      1. Brandon Burns

        You’re testing your copy, so I’m sure you’re optimizing. Still, I’d think about something that clearly explains what it is I’m downloading.In the part up top where it says “Give yourself the same tools…” I’d think about something like “Our WordPress plugin allows you to test multiple headlines and know which one will get you the most readers.” With a button that says “Download the Plugin”Not saying that’s exactly it, but something that clearly states what the customer is getting. And that’s the end of my two cents. πŸ™‚

        1. LE

          Agree. Bullet points might work well with stating multiple benefits.

      2. LE

        Shana – I’m not a fan at all of any site that tries to hide some basic pricing reference. I think you need price points beyond “hobbyist = free” and “deep pockets = call us”. Which appears only if you click the sign up at the bottom and not at the top (assume that’s a link that needs to be fixed?).There has to be a reference price wise of what people are getting involved in. And not only that but the price reference should be right on the home page. There should be a pricing link right at the top.You could always shorten the name to baywitch.com by the way. Or witchian.com or even better, and sure to get attention:bitchian.com !(Btw you need to get baysian as a typo of bayesian …)

        1. ShanaC

          we know this. its related to incorporation problems. The state of delaware is a fun place to call…I assure you πŸ˜‰

          1. LE

            Not clear if you are talking about my comments relative to the name or with regards to the pricing? Either way I don’t see how incorporation issues could be involved in either. Separately, you can be XYZ corp and trade under “Shul Labs”. (Which is open by the way…)Jordana’s name, bio, and icon, by the way wasn’t far off enough to scream “actually is the office dog” which is why I thought it was a real person. I actually googled it.

          2. ShanaC

            Relative to pricing.The long term goal is to get the mascot to be the customer success person. We’ll see how that goes

  21. Salt Shaker

    Bandit testing is far, far superior to A/B testing, a conversion tool favored by many publishers today. Since Bandit is a bit more complex to explain than A/B testing, it might be beneficial to include a couple of use cases (and testimonials) off of your home page. Bandit can be both more effective and efficient than A/B testing.Wish you much success w/ this venture…Certainly addresses a legit need.

    1. ShanaC

      We radially shrunk the product for basically this reason. Keeping it to headlines πŸ™‚

      1. Salt Shaker

        I hear you, but are your headlines perhaps a bit too vague? (I’m kind of in Brandon’s camp here.)Every publisher understands and likely has experimented w/ A/B testing, while Bandit testing delivers a higher level of optimization.Why not position your company comparatively to A/B testing? A/B testing is like The Flintstones, while Bandit testing is like The Jetsons.”Still relying on A/B testing?… Upgrade your optimization tools w/ BW to enhance readership.”…..Just saying. Love the name, btw.

        1. ShanaC

          Small publishers tend to be weaker in that regard. if you are big, then they might – but you’d be surprised how many large publishers don’t understand the idea of a two tailed test, for example.

  22. awaldstein

    Hey ShanaTwo requests:1. How does this work to the extent you want to share? I this SEO based at its core?2. Many bloggers should be interested but their traffic sample is low–maybe 1-5K per post. How large a sample is needed?

    1. ShanaC

      1) No, it is not SEO based at its core. You may use SEO to get people into your blog, but now that they are there, how do you get them not to push the back button and read more articles? You show them more potential cotnet – usually a title, and in some cases images as well. But what is the right title to make them stay and read another article? And what is the right title to make them share an article? (that’s the question we’re asking)2) The minimum for testing two titles is 350-400 pageviews. While the test won’t “finish”* you’ll see uplift nonetheless. So our minimums are way smaller than the bloggers you are talking about. And it is designed that way on purpose.*Its not designed to really finish, it will react to traffic as it continues to happen.

      1. awaldstein

        I like!If you are selling this to the long tail of bloggers out there you have some interesting work to make it really simple to understand.Looking forward to your blog posts!

        1. ShanaC

          what would you suggest as a starting point?

          1. LE

            “what would you suggest as a starting point?”I think you need to get a bunch of more or less clueless people [1] sprinkled in with some more knowledgeable people to review your marketing content and what you are offering. You need both groups to understand what you are selling. You aren’t either person you know to much already.The headline might be: “Why stupid people can help you with yourmarketing and smart people can’t”[1] Of different levels of intelligence.

          2. ShanaC

            we’re trying that out actually. Its why we nixed the data science taglines. Most people felt overwhelmed by data science, or that they would lose control.

          3. awaldstein

            Not a believer in asking your market what to do.Where to start? The beginning.Your sense of the value of the product to a segment of the market is where to start.And wherever that is–be it me or someone more or less technical, make it clear, make the choice easy.

      2. LE

        Arnold: “How does this work to the extent”I think you need to put more on the site to this end. I think some people know nothing and just trust in a marketing message. Others who know more won’t believe in something unless they have some clue as far as (in non marketing speak) what is going on under the hood. Otherwise they make assumptions (because they know something) and those assumptions might be wrong.Part of this is the curse of knowing to much. If you know little you tend to take things on faith sometimes. If you know more you tend to want to know how something is possible (because you have a frame of reference).Here’s an example. If you told some people “so with this app you can walk into Starbucks and they will have your drink ready” they might just think “geez you can do anything with an app that’s great!!!”. But if you told me that I’d want to know how that was being done and how it was even possible. And I might actually miss a few great features of the product because I might assume that it might implement a way of providing that benefit that I wouldn’t approve of.I don’t think you need to clutter and confuse the first group of people just make sure the info is available for the latter. Short blurbs on homepage and more detail in a faq.

        1. awaldstein

          Sorry, not following.

          1. LE

            I was just attributing the quote to you.( I wasn’t asking you a question.)Or do you mean that you don’t understand the points that I am making?

          2. awaldstein

            none of it honestly, inlcluding the analogy between a starbucks card and a WP plugin.this is a very complex product.tactics are kinda useless till they figure out what it is and who will buy it.

  23. Elia Freedman

    Yeah, Shana! Hope it does really well.

  24. Dale Allyn

    Congrats, Shana. Hope you feel better soon!

    1. ShanaC

      i can tell you I do not recommend doing a 7+ mile hike with bronchitis…

  25. JLM

    .Congratulations, Shana. Good luck and crush it.JLM.

  26. jason wright

    how about “Google Bitch”?the web is becoming ever more tabloid.

  27. Paul Shapiro

    I’m currently using Noah Kagan’s KingSumo headlines: https://kingsumo.com/apps/h… It’s the same idea and I like it very much πŸ™‚

    1. ShanaC

      its not – we’re tracking other things for a better optimization experience. My cofounder is also probably the lead person in the US for implementing these kinds of optimizations.

  28. Pete Griffiths

    Rename it ‘Trollbitch’ ??

  29. ShanaC

    i’m in colorado recovering from bronchitis/really bad chest cold.

  30. LE

    Never know what you might learn :)Learning is not always good. Sometimes it can mess with your gut. I never have read any books on negotiation and never plan to. I don’t want my brain messed up with other people’s methods or what works for them. I have something that works for me that was developed over time. Not interested in something that will derail my gut.My guess is that Fred likes AVC “just the way you are”. More visitors might mess with the delicate balance between regulars and drive by visitors who could ruin the atmosphere. Then next thing you know everything changes. Not to mention that he’s probably tapped out with the amount of commentary there is now. He has a day job. It’s not a hobby.I would think that the product is good for larger sites that have to churn out many headlines and make money primarily by advertising as opposed Fred’s use as business development and to talk about what is on his mind each day.Personally I would think it’s fairly easy to reverse engineer what makes a good linkbait headline. I mean if you keep your eyes open and pay attention (to what others are doing) it’s kind of obvious.Fred could have just said “Why I don’t do A/B testing” if he wanted to get more people interested in the post for example.