AVC Down Time?
Yet another post about something I supposedly don’t actually care about 🙂
I am pursuing a theory that the cause of the AVC down time that I posted about last week has to do with a burst of traffic from social media (mostly Twitter) that happens right after I publish a post on AVC.
I almost always experience this downtime in the 5-10 minute period right after I post. That is also frankly the time when I pay most attention to AVC each day. After that I tend to move on to other things.
So I am curious. For those of you who have experienced this downtime (I realize many of you have not), do you recall when it typically happens to you?
Comments (Archived):
i direct navigate and haven’t noticed any downtime
I usually see your posts pretty early — via feedly – as you said – downtime, when it happens usually seems to be pretty close to your posting time.
This has only happened to me once or twice, and I usually do look in the early am…and usually hitting “refresh” solves it.
Yes, in the AM typically for me. But then again I typically don’t check later in the day unless I have a reply to do so…
It’s always up for me. Should I be concerned if it’s up for more than 4 hours straight?
Yes visit ER if lasting more than 4 hours. I always wondered about that “4 hour” time frame. Presumably they are taking into account the time to decide to go see a doctor as well as the time to get to the hospital and time waiting in the ER and so on. My guess is that damage doesn’t begin until 8 hours.
“You take the blue pill, the story ends. You wake up in your bed and believe whatever you want to believe.”
A suggestion to verify this theory more precisely than by testing our memory:Pick tomorrow as a test date for example, and commit to publishing at a specific time, eg. 7am. Then, a number of us will try to go there exactly between 7am and 7:10am, and we can report with more accuracy for you. (even better, link to a Twitter poll to collect the data)
Or, just sign up for a monitoring service which will do the checks for you on a continuing basis.
LE:Are you suggesting something simple? How dare you.Too much energy spend on the unnecessary when there are necessary events occurring. Gheez
A quick and dirty shell script using curl that anyone can run from the command line (hopefully not to many people).while sleep 60doCONNECT=5MAXTIME=10DATE=`date +%Y-%m-%d_%H:%M:%S`echo ${DATE} >> avc.logecho “————————” >> avc.logtime curl -vk –connect-timeout ${CONNECT} –max-time ${MAXTIME} http://avc.com/ | tee -a avc.logecho “————————” >> avc.logdoneThe connect and timeout say that a connection should take no longer than 5 seconds and the whole shebang should only take 10 seconds. Those can be toggled obviously.
Morning. Also seems like it’s 2016 and if BlueHost can’t support you it’s time to move providers? DreamHost has been great and cheap for me.
has to do with a burst of traffic from social media (mostly Twitter) that happens right after I publish a post on AVCPretty easy to prove this out with logs obviously. That said this is a static site. I can’t imagine other than a DOS why this should happen with any burst of traffic from social media. The host is throttling you to get you to upgrade your account.
i’m in Europe (GMT+1) and i don’t experience the downtime you are observing, and i generally get to see your latest post before most people in the US (unless they’re in California and still awake).is it a mirroring issue?
Same here. I see most posts during the week before they have many comments and it is never down. Don’t follow on Twitter necessarily (I mean, I *do* follow, just don’t notice the post there).
I have a notification to your avc account when it tweets. That’s when I read your posts and mostly witness the downtime.
so it could be a traffic surge prompted by the tweet notification, as Fred suspects?birds of a feather do flock together.
You down with A V C?Ya you know me!
Another thing to consider is that you are using a redirect such that if someone types in http://www.avc.com they hit the server and it tells them to go to avc.com via a 301. This is taking up resources and ideally should be eliminated.https://varvy.com/pagespeed……
That 301 Title tag shows up as the page content when I share links to the blog on Facebook. It would be great if it was fixed.
I’m so excited. i’ve been doing Codecademy and can actually read that! (Well, most of it.)
I often read when your @avc handle first tweets in the mornings and have no issue seeing the posts.
the irony of this post is that we now know that getting to read a new AVC post in the first 5-10 minutes after its publication is critical for direct communion with Fred, which will amplify the problem of downtime. delicious.it feels like the paradox of quantum mechanics and the wooden box. opening its lid changes the state inside. observing is not a passive activity.i don’t know why i’m writing this because he’s already long gone.he does care really.
Perhaps you should host with someone where this doesn’t happen.
If you want to verify it – you could first set up a monitoring service (you could run a free-trial of pingdom and it takes 5 minutes to set up – not related to the service, just a former satisfied user). Then you will know when and how often AVC is down.If your hypothesis is verified, you could run some load testing – to really get a sense of after how many users do you start having the issue.
I’m more inconvenienced by the fact that three posts have taken up what otherwise would have probably been more interesting topics 😉
Never have issues
I am facing since last few days. I thought it’s DDOS attack on AVC.
Your theory seems like it might be plausible. I very rarely have any problems loading AVC.com. I rarely use Twitter and I do not have a Twitter account. (To me Twitter and Facebook are like television and radio, mostly a waste of time).Why not create a URL called, perhaps, AVCbeta.com so that you could do A to B testing? Your AVC.com regulars who do experience problems loading AVC.com could load AVCbeta.com when they have a problem with AVC.com. You might put a 5 second delay on loading AVCbeta.com to dissuade users from using it as their primary URL for viewing content you post on AVC.com.
I don’t post right away but then I don’t go to first run movies on opening night either :)Regardless though your people should have metrics, a root cause and a path to solution.
I follow @avc on Twitter to remind myself to check your site daily. But some days you forget to tweet about a new post. Have you ever run an experiment to see if your problem happens only on days when you tweet about a new post?
On another note, I am amazed the Mets have taken as much punishment with injuries as they have and are still in the playoff race. Fred rides his bike to work-and so does Cubs player Ben Zobrist. http://cltv.com/2016/09/18/…
Never experienced it. I am sure you have a massive list of loyal email subscribers. If you have not already, may be consider delaying the email delivery. Will help reduce the peak traffic volume.
Have never had this problem. But unless I am traveling, pulled an all-nighter (fewer these days) or had to be up at dawn or you are traveling, I am reading your post hours after it went up. The benefit is that by then quite a few comments.I do appreciate that you now post from the West Coast a few months a year.
Now if we can just get him to Chicago for a bit…
Haha. Good luck with that.
So, right, for a first cut, there are three steps:(1) Detection. Monitor continually in real-time to detect anything might call an anomaly. Here there are two ways to be wrong: False alarms and missed detections. Really, the work is necessarily essentially some continually applied statistical hypothesis tests (see introductory or intermediate texts on statistics). There a false alarm is called a Type I error or the significance level of the test. Basically, false alarms are the price pay for the detections. So, want to be able to adjust the rate of false alarms and, to get the most detections of real problems, take the highest rate are willing to tolerate. Then for that rate of false alarms, different statistical tests can differ greatly on the rate of detections of real problems. Of course, want the test that, for the given false alarm rate, will give the highest detection rate; a way to find the best such test is in a classic result of Neyman-Pearson. I have an especially general proof based on the Hahn decomposition. The computing can result in a knapsack problem, that is, in NP-complete. In a sense, the monitoring is a first filter to keep down effort in (2) diagnosis.(2) Diagnosis. Given an alarm from the monitoring, want to do diagnosis, that is, determine if the anomaly/alarm is a detection of a real problem or a false alarm and we want to find a cause. Here we might have some candidate causes and for each may want to run, say, another statistical hypothesis test to select the ones to investigate.(3) Correction. Once have a good diagnosis, want to do correction, that is, solve the problem. Then will want to record the data that gave the detection along with the cause and also the correction and, then, use that data to start to automate diagnosis and correction. This is done in parts of computer security where we detect a computer virus, diagnosis the virus, collect data and find a signature of the virus, and in the future use that signature to automate detection and diagnosis.So, we are into some statistical hypothesis testing. There is a large literature for that, an example of more glories in the QA section of research libraries.Of course, computer monitoring is not the only case of (1)-(3) — we can get involved in much the same in monitoring a car, airplane, instances of industrial or production equipment, the electric power grid, other systems of wide variety. And of course much of the work in medicine is close to (1)-(3).In this case, our candidate cause is something from Twitter. So, we would like a statistical hypothesis test for that.Exercise: Find a good statistical hypothesis test for considering if the cause was load from Twitter.With all this work on statistical hypothesis tests and automation of the work, the resulting (1)-(3) starts to look smart. So, maybe some people would say that this work is part of artificial intelligence and/or machine learning. Alas, I see the work as some applied math, and from all I have seen of AI and ML they are still not close to statistical hypothesis testing.The real world and in particular computing in practice are still a long way from doing well on (1)-(3). Are there startup opportunities there? Maybe:On the one hand, sometimes the costs of real problems going undetected too long can be large, large enough seriously to hurt a major company. So, there is some motivation for getting solutions.On the other hand, there are some obstacles:(A) For anything on the way to a comprehensive solution, there can be a lot of infrastructure software to develop, document, install, teach, and execute. E.g., to be comprehensive, in this case, would have to be prepared to have and manipulate some Twitter data.(B) Since the amounts of data readily available are large beyond belief, there can be a lot of effort in data handling.(C) So far management is not very clear on (1)-(3), e.g., not nearly enough people took a course in statistics that had good coverage of statistical hypothesis tests.What do people do in practice?(A) Start with products that have good reputations for reliability.(B) For well known problems, apply existing, respected solutions.(C) Otherwise, detect problems from somewhat obvious symptoms and diagnose and correct those problems one at a time. With the corrections in place, hope that the problems don’t recur. These problems and causes may be too rare to be worth automating.So, don’t really try to automate (1)-(3).Net, to me, in the basket of startup opportunities, this one looks not so good.
I’m guessing you pay attention to your own blog after hitting publish because of the commenting that boils up once we read your almost always excellent thoughts. Why not move to a follower and messaging app model?You should invite all of us to a WeChat discussion group and post your thoughts as a blog post within the messaging app, and then engage with us there in realtime or delayed time.You can even use third party apps to put together a radio show or podcast, and I am sure there are many English speaking Chinese investors and entrepreneurs who would want to read or hear your thoughts.I know this because I have used some of your thinking in a course that I have held in beijing during the summer for Chinese-speaking students of STEM and entrepreneurship:Disclosure: tooting my own horn about a China-based entrepreneurship and high tech learning program we run with folks at baidu and some other companies in Beijing.https://vimeo.com/178184499…
Douglas Crets:is your post considered a shameless plug?Was the Albert Hu of Asengua Ventures allowed discussion?
It’s definitely considered a shameless plug, which is why I offered a disclosure. I am not sure who Albert Hu is, and if you think I should know him, please offer an introduction. Thank you!
Douglas Crets:You should not want to know Albert Hu he is a US Federal prisoner. Google him with Fortune Magazine as a tag.
What is unnecessary has become necessary. And the necessary has become unnecessary. Just get her done! It can not be that complicated for this bright, well-educated and well-refined group. There are some serious things going on. If resources were an issue that would understandable.This can’t go from being the sitcom MASH to The Secret Diary of Desmond Pfeiffer (UPN, 1998).Various issues/concerns/problems were cited. LE provided two workable solutions.In other news.
Thats the beauty of reading it in Feedly – Now if only I could comment from there too….
I read it in Feedly, too! The slowness of Disqus on mobile sometimes prevents me from commenting, but I’ve very rarely experienced AVC downtime.
It’s great to gather anecdotal reports. They can help put a face on a problem.But better to monitor.GA has realtime, which you can monitor right as and just after you post.http://uptimerobot.com can help & is free. Pingdom.com is a paid service. There are many others. These will give you notification at the time of outage, and historical graphs of when or where outages occur. They may be localized to certain regions based on where your servers are hosted, where your readers are based, network latency, and many other factors.
Utterly random – mid day, early morning, late at night.:/