The Behavior Of Your Users Normally Doesn't Change Overnight
A few weeks ago the traffic coming to the new usv.com dropped off by 20 or 30% week over week. Brian and Nick were wondering if it was related to the new design we rolled out at the end of January. They decided to take a deep dive into our analytics to see what was going on.
I told them it had to be some sort of plumbing issue. Something that was hooked up to usv.com must have gotten unhooked. Because I have rarely seen the behavior of an entire user base change drastically overnight because of something like a redesign. Change can come pretty quickly, but in my experience it is months not weeks or days for something to drastically change in your user base resulting from a subtle product change/tweak.
They noticed in the analytics that front page views were steady but traffic to the article pages was off by a lot. I suggested that something in our twitter plumbing was off. And sure enough, we figured out that the autoposting of popular articles on usv.com to this twitter handle was broken (it is actually an RSS issue that is being fixed today).
I tell this story because we all encounter this sort of thing along the way of building and launching and growing a product. We make tweaks and something changes right away. That immediate change is usually related to something that brought traffic (google, twitter, rss, email, appstore) and not a design change. More gradual changes (up or down) are usually because of design changes.
There’s a difference between these two kinds of effects and it is important to understand that.
Comments (Archived):
True…and worthwhile to mention the other side of the coin, where a design change can sometime impact behavior, such as bounce or conversion, more quickly.
Yeah, seems entirely possible that you could have a massive drop off in traffic in a week based on design changes. You can practically type in a random domain name and find a company claiming how their big data-driven A/B testing solution increases traffic rates by 20% to 30%. The converse should be true.
Ha! I experienced this in the RSS by being redirected to the sources directly… i found that a bit annoying because it meant I could not upvote/comment directly, but I also kind of liked that USV was not ‘hijacking’ the traffic from the sources like some RSS readers do for example…
Still had to be freaky for Brian & Nick
yeah, really confusing. but we figured it out. now we need to fix it
Very much like your fine work!
True with many things in life … progress is usually incremental and only takes dramatic swings when drastic action is taken. Relationships, careers…
you are going all philosophical on us this morning Jimgiven that you are the resident AVC comedian, i am wondering if you are going to stand up with the comedians on Disqus comedy night?
So….a Troll, a Spammer, and a VC walk into a bar…
What’s the difference?
That’s what the troll said.
ouch! never drink and invest.
Koolaid or other beverage or neither?
Disqus comedy night? an internal thing?
A New York thing I believe.
every day at disqus is a comedy. cc @wmoug:disqus @fredwilson:disqus
Formal request for video of Daniel doing standup please!
Fred… You’re now a technical analyst!
Henri Cartier-Bresson talked of the ‘decisive moment’. see it for what it is, and act upon it. his was to press the shutter release button. i think buttons are all around us.
Indeed…http://mediastore4.magnumph…
a classic. his legacy will last.
That was one of only 2 HCB photos that were cropped — he usually insisted the negative carriers be filed out produce a thin black border around the frame as proof. A master of composition.
Dramatic swing aka fuck ups? 🙂
That’s one way.
hey, jim. hate to hijack your thread! but i’m looking for more info on ad buying with disqus (getting content placed in the related links section). the site is slim on info, and i’m too impatient to wait for someone to respond to the “request info” form. i’m specifically looking for CPC costs (or however you guys charge), and any case studies showing effectiveness (ecomm preferred).help? brandon.at.wanderandtrade.d…
emailing you now
I dunno I feel like I got kicked out of college pretty quickly
I’m sure there’s more to that story and what led up to your uninvitation (yeah, I made that up) from college. Let’s start with your pre-K years, shall we?
got sent home first day of K because my teacher couldn’t hear well. i stand by that opinion.
There is another plumbing issue related to Disqus. If I’m the original user that posted the link & other users start to comment on that post, I’m not getting notified in my Disqus Dashboard or via email.
yes. we should figure out why that is not happening and address it. on the list.
that would be a very nice fix. Having to actively check a website for activity seems very pre-notification. For all of the anger at push notifications, they really are very helpful (even some email, like in this case) when people have total control over how much access to give them.It would also be nice to at least get a Disqus dashboard notification for responses to responses. In this case, Will might like to know this comment happened without actively scanning his past posts. Maybe some power users would find that overwhelming, but I have almost missed high value comments.
learning the lessons of a startup by acting like a startup
Look, I know it’s hard to fire the person who got you this far, but if you’re gonna scale this thing, you need to move that guy aside and hire this lifelong friend of mine who was Cisco’s CTO, okay?My preferred shares say you should do it today.
IMHO firing the guy for forgetting one pipe during the major reconstruction (which redesign definitely is) isn’t a great idea. As usual the statistics should be taken into consideration.At least they know that it could be a good idea to configure alerts / monitoring / better statistics on traffic sources. Next time USVs could be analyzing why traffic spiked from Twitter related to the new article and today’s issue will make them ready for that kind of analysis (instead of simply contemplating “we are doing a great job – our traffic grows”).
My comment was a parody about how bad VCs often behave.
Ooops 🙂 I see now 🙂
RSS = Really Slippery Slope
You’ll get this problem figured out. And then it is a never ending process of improving bounce rates and conversion. Great article by Bill Gurley on the subject: http://abovethecrowd.com/20…
right. like grinding in golf.
Love the AVC conversion goal btw!Host the most interesting tech/ startup conversation +Share your thoughts PLUS those of others+Make people aware of services you have backed+State often what you are looking for= talk to start ups who know you, like and trust you, and know what you are looking forConversion!
exactly
http://101.ru/?an=personal&…
Whatever that is. Internet not saying much [1] about that other than the obvious usage of “grind”.I remember reading Ray Kroc’s book about McDonalds that was called “Grinding it Out”. (It’s about real estate and Harry Sonneborn was the master of that strategy.)[1] http://golftips.golfsmith.c…
Hehe, that is so real life! How about this… My friends worked for a startup that experienced a monthly revenue drop from $30M to $20M. A pretty drastic drop if you ask me. No one from sales, marketing, PM, CEO to developers could figure out what has happened for a few months! No kidding, a few months… They kinda gave up saying well, sh*# happens, our users have lost interest. Then my friend was doing some consulting job for them and discovered that one server out of three wouldn’t accept incoming traffic, so it was all bounced. A $10M/month discovery:)
Made a fundamental error attributing one thing to another thing. (Didn’t look for the Zebras.)For example “big snow storm and the phone isn’t being answered”. You don’t assume that the person who had to open the place got whacked the night before and you don’t assume the place burned down. You correlate the most obvious event and say “ok that’s why” and you don’t look any further.Anyway the thing is with proper monitoring the situation you describe (30m to 20m drop) would be trivial to uncover. An entire machine not taking traffic? Monitoring would pick that up in an instant.Logs would have shown that. Don’t have logs of everything going on? Major screw up. Verbose log everything and anything. Disk space is cheap. This isn’t the 60’s.By the way strange things like this (not this but like this) are a regular occurrence for any decent sys admin. [1] They like figuring out real life mysteries like this.They kinda gave up saying well, sh*# happens, our users have lost interest.Well good for them they deserve to fail with that attitude. I hope they had fun playing their video games though.[1] A developer (and obviously a PM/CEO) is not a sys admin. Although they could be.
LE that was exactly my reaction. How stuff like that wasn’t detected by monitoring? Who knows…it’s easy to judge post-mortem. However, the biggest screw up is not the monitoring though but the fact no one followed their common sense and tried to replicate the failure, channel by channel. Instead it was all wrapped up as a blame on the sales team, ads ineffectiveness and poor marketing. Dah, things like that don’t happen overnight as Fred pointed out.Many times looking at my own analytics I would discover certain funnels drop conversion let’s say after a page re-design. The most natural thing would be to play a usability blame game. But after looking more carefully and analyzing users profiles (thx GoogleAnalytics, Kissmetrics and such…) we would discover Oh, our Continue button isn’t functioning for Safari users etc. I believe that 10%-30% drop could be attributed to some usability/design implementation but when it’s way above 30% chances are someone has to look for a simple technical glitch.
.The ability to “walk the cat backwards” as a means of exploring what has gone wrong is a real skill. Not everyone recognizes it and damn few people know how to do it.Perfect example.Well played.JLM.
Recently, after many years, I got to the point where I could fly the rc helicopter backwards in circles.Anyway with respect to “walk the cat backwards” that seems to be the rough equivalent of “genie back in the bottle” [1] or “unring the bell”.[1] Hey remember her, “yes master”?
.No, the cat leads you to the scene of the crime but does not make the pain go away.I used to love Genie.JLM.
PantherKitty loved your phrase JLM. I think she wants to make a video of the cat walking backwards to demonstrate. Artist in training.
this is one of my favorite things to do. i like to debug way more than i like to program.
Once you remove the obvious, as Sherlock would say…
That’s not always an appreciated talent. It’s akin to saying that I prefer to criticize what you’re doing instead of doing it. Constructive criticism is a very rare talent, at least from my observations.
As a QA Engineer I appreciate this comment as well.It’s an odd relationship that QA & Dev has & is one that requires a near-perfect balance to function remotely well. Maintaining that balance while providing criticism is a unique, incredible, and rewarding challenge.I find that genuineness, humility, and vulnerability are key to the success of this relationship.
As a QA Engineer I appreciate this comment.
that’s exactly my point
You should also not have exponential changes caused by a constant variable.If that Twitter function provided exponential week over week growth, then it could be the sole issue. But if it provided a relatively constant X% of traffic per week, the drop should have been a similar constant number (probably higher due to halo effect) and, thus, I’d keep looking for other reasons for the exponential loss.Also, conversion to articles from the homepage and bounce rate from it will easily tell you if the redesign resonates or not, much more than the traffic numbers which, with USV.com functioning as a media channel where 90% of traffic is likely to come from the spread of content, are likely to revolve around the quality of the content more than the platfrom itself (save for changes in ease of sharing the content). Also, change in returning visitors is a good indicator.UX is a skill and a specialty. Having someone in that capacity will help you more than anything.
yeahthe RSS / Twitter feed brought in a constant (and linearly growing) amount of traffic. it was a significant % of traffic, so there was a major drop to the post pagestraffic to the homepage was constantconversion from the homepage to post page is a more interesting one. a large part of the redesign is that you don’t have to leave the homepage to consume the content, so we’re still adjusting what that meansand of course, quality of contents is what trumps all
sounds like you guys have a good handle on things. 🙂
This fits with my thought that the core people using USV at the moment are the evangelists, people who are there because of who USV is and who USV attracts – and so the content is presumed to be of quality. So the traffic drop was related to those core users using the Twitter feed as reminder for the content.
I’m assuming some here, but after looking at @usvconversation’s timeline it seems it stopped tweeting for two weeks. If it didn’t tweet and nobody noticed or alerted you that it stopped, that seems like the bigger issue.
Especially important when managing expectations of management as product manager
Fredj here is teaching all of us how to act like all-stars (Jeter, NYC is gonna miss u!) All-stars have the experience and capacity to not only stomach an unusual occurrence like the one mentioned in today’s post, but also the knowledge to know that something is broken and the courage to take quick actions to fix it. That’s a 5-4-3 double play!
People are primarily at USV.com for the presumed quality content because of the core group of people running it and the core crowd they’ve attracted.It leads to the question – does design really matter if what you’re providing a person fulfills their need?In reality, the answer is yes, though not for the 80+ percentile. You can’t really tell unless you A/B test between a large base of users either – and even then it’s generally a bad idea IMHO, except for a few scenarios.This is why feeling and intuition need to, should, come into design and planning – because then you’re building in the eyes of the visionary – based on their observation of the world, and reflecting it out into the physically built world.
It leads to the question – does design really matter if what you’re providing a person fulfills their need?When dating you usually try to make a good impression on the first date. In order to allow someone to get to know you. So that they stick around. [1]This is really the same with design on a website. Put your best foot forward in order to have the greatest chance of success. [2]Having good design will help you attract (meaning they won’t bounce) people that may stick around if they like what they see (content wise). It is certainly less important (to a degree of course) with the existing user base. So as Fred noted it is unlikely to be a cause of a drop in traffic (but still possible).Note that when people go out to dine they want both good food and a good atmosphere and environment [3]. Now of course if your food is just so super special you can probably pack the house even with fluorescent lighting and pizza restaurant seats. But you don’t see that happening much, do you?[1] Later it doesn’t matter as much as any person who is married or in a relationship knows.[2] I get into this back and forth with Arnold re: domains – my point always being you need as much of an advantage as you can. Can you succeed without a good site design or a good domain? Of course you can but why not have every advantage that you can?[3] Dining of course is entertainment as much as eating so this makes sense.
This is something a good and well-documented QA process would have caught, or maybe tests; It’s why it’s a role that exists. 🙂
@fredwilson:disqus for long past now, I have been relying upon the AVC ‘musings’ RSS feed via my desktop Outlook 2010 client so, having read your comments above, decided just to check the delivered ‘musings’ thread and happy to see that my RSS stream to this forum has been uninterrupted by any changes to usv.com at your end, granted that the changes to the usv.com site are not related to the presentation of this forum but you can never be too complacent where anything IT is involved. :)That being said, however, what is REALLY interesting about this subject is that a) a sudden drop off in traffic can be so quickly observed and b) that analytics can so quickly help discover the possible source of the problem and c) that a broken link to twitter – of all communication paths – could be the culprit! Who could have guessed that the use of, or inadvertent non-use of, a ‘twitter’ stream could account for such a change in traffic? That last part alone is very insightful information, is it not?
yes. but sometimes, just sometimes….[contact list import + send game challenges by email]#nostalgic
.This example highlights a skill that is widely unrecognized and misunderstood the ability to:WALK THE CAT BACKWARDSThis skill is essential to the intelligence business wherein a result is not consistent with what someone knows to be true. It relies upon instinct.When Amb Susan Rice offered the explanation that the Benghazi assault had been the result of some vexed movie critics anyone with a brain immediately jettisoned that lie — movie critics do not know how to use mortars, RPGs, crew served automatic weapons and infantry tactics.As you began to walk that particular cat backwards, it became immediately apparent that the absence of a spokesman from Def, CIA, State created reasonable suspicions. Once it was revealed they were working on the “talking points” together, the cat was out of the bag. Where the cat went thereafter was a mess that is still not resolved.In applying the same notion to l’affaire Snowden, there is a huge body of instinct that has now become known to be the truth including something as small and obvious as the SOB purloined others’ passwords.The big lesson with Snowden is how lax the information was guarded INTERNALLY. This was a classic inside job. The pros figured it out quickly.With technology and BMWs out of warranty, the ability to walk the cat backwards and to apply the most logical and inexpensive solutions is the key to survival. Don’t replace the engine as your first fix.Funny, not many people get the concept.JLM.
Don’t replace the engine as your first fix.Grump gramps rant of the snow day. (As opposed to sno den).The result of a few big changes over the years with products (in general).1) Shit was more expensive back then and you fixed it you didn’t even think of replacing. (Was also not available overnight from Amazon.) [2]There were people called “repair men”. Remember Dad telling you not to play with the “electric windows”? I do. “You’ll break it” he said. My Dad had a tube tester [1] and I used to test tubes (that said I’m not that old btw but I do remember doing that). For that matter he gave me .25c per pair to shine his shoes.2) Said shit was mechanical and not electronic. Advent of “solid state”. So you could fix it. In my first business I owned equipment that was primarily mechanical. You could hear, see and feel how to get it running. Later it became computerized in many aspects. Not the same thing. Much different skill set and tools needed, typically just replace the circuit board.3) No such thing as “bumper to bumper” warranty on cars. Certainly no such thing (as with BMW) as “we even replace the brakes and wipers”.4) No such thing as big box stores that you return anything and everything to. Used to be “Mr. Shop Owner”. No way you are putting anything by on him. Don’t like the tv you bought? Go fuck yourself it’s yours. (Remember those “you break it you buy it signs”? I remember getting yelled at by a shop owner for touching something. Today that would get someone arrested, right?Oh, so my point? We are not back in the days of the white shirt guys in mission control at Nasa during the space launch. Sure they drank and smoked but they took the job really really seriously. Today it’s like the world of the collective slacker screw up.Anyway it’s a whole new ball game Callahan.[1] Attached. It’s for sale if anyone is interested. Apparently musicians use these things for something.[2] I grew up in business keeping all sorts of spare parts and extra equipment. Took me a while to figure out that anything I need can now behad overnight from Amazon or Ebay if I can wait that long (sometimes you can’t of course). Fundamental error in not keeping track of how things change and staying the course.
Fred, do you get feedback metrics on how many people are opening your e-mail (therefore probably reading) through your RSS feed?
i do but i dont check them that often. its about as many as visit the website each day. rss is about that same amount too
That’s the reason why Google Analytics and all other mature analytics systems have “Alerts” feature. You could configure a threshold after which you get a notification. This helps to take an action sooner.Overnight increase usually means someone huge walked into your party and told his friends. Analytics will help you identify that person and get more results than simply a traffic.Overnight drop usually means bugs and issues.There are some exceptions though. A viral growth when something really smart was implemented as example. But more like is the change in third party logic. A.e. Google Play / AppStore ranking mechanism change could impose severe fluctuations. Google Search / Facebook Timeline changes could spike / burn your traffic in a second with their algorithms when they decide what to show in your search results / feed.
What I find most interesting about all this is the size of the source of traffic from Twitter. 20-30% of traffic from automated Twitter posts seems quite high to me, but I’ll not argue with Fred’s data.@fredwilson:disqus, if you don’t mind sharing up to date numbers, how much AVC traffic is organic/Disqus-based, search, Twitter, and other traffic sources?I know that my traffic to AVC is usually organic (I drop by frequently) or from a Disqus digest of people I follow (stalk?)…
So how can you tell if it *is* because of a design change? What should we be on the lookout for?
Really Spectacular Syndication 😉
You Rambunctious Smart Soul
Really Silly Sequiturs