Disaster and the Cloud
About a dozen years ago, the Gotham Gal and I purchased a townhouse in Greenwich Village and did a gut renovation on it. Down in the basement, we built a large data room. There were racks and racks of equipment in this room. There was a home audio system with mp3 servers. There were network attached storage devices for family photos and videos. There was even an exchange server that I had taken from the offices of Flatiron Partners which ran our mail and related systems. It was an impressive room that would raise oohs and aahs from geeks who like that sort of thing. When we sold that house in 2007, we decided to decommission all of that equipment and put everything into the cloud.
The home audio system became Sonos units connected to Rhapsody, last.fm, and a number of other cloud based music systems that were available at the time. The family photos and videos went into cloud based file storage systems. The exchange server was scrapped and we all got onto Google Apps.
I thought of that room when we went back to visit our building on Tuesday in the aftermath of the storm. Our basement was filled with the Hudson River and I shivered to think about that townhouse data room filled with water to the ceiling. Of course that never happened. But what if it did?
We do have a data room in the basement of our building which was filled to the ceiling with water this week. But that data room mostly has network switches, routers, cable and DSL modems, and patch panels in it. We have redundant data networks in our home. When we get our basement cleaned up and dry, we will pull all of these switches, routers, amd modems out and replace them with new ones. And we will be back up and running. Because all of our systems are now up in the cloud. And I know they are operating fine. Because all week while we have been displaced from our home, I have been accessing all of them from our friend's home.
But it goes even deeper than that. For the next couple months, the Gotham Gal and I and our son will be living somewhere else. I have a home office that for over twenty years has had a windows desktop machine running Quicken on it. I am a bit obsessive about keeping good financial records and files. I thought briefly this week, "how am I going to manage all this stuff while we are displaced?" But then I remembered that we moved everything to Quickbooks Online a while back and I can make any place my home office because my data is in the cloud now.
And this applies to our portfolio companies as well. One of the companies I work with went down this week because their primary data center was flooded and has not come back online yet. Their engineering team worked pretty much around the clock to move their primary application to Amazon's cloud so they could get back up and running. As the CEO and I were emailing about this disaster, I mentioned that one good outcome of all of this is that our application is now redundant. We can run it in a data center and/or the cloud. Which of course is a good thing.
Whether it is a company or a family, moving your applications and your data to the cloud is a great disaster preparation effort. Most companies deal with disaster planning and redundancy when they "grow up". But many families do not. This week was an eye opening experience for me about the value of putting your applications and data in the cloud. We did that a while back and it has made being flooded and displaced a lot easier. I cringe to think about what would happen if we had replicated that big data room in the basement of our building. Thankfully we did not.
Data rooms to the penthouse, swimming pools to the basementThe Cub Scout motto: Be Prepared
I think that is the boy scout motto. I am proud go say that I am an Eagle Scout
In this age of gender equality girls are in too.I was a Sixer. My fab five were a great team.
Correct…the Cub Scout motto is “Do your best!” (I’ve been the Cubmaster for my boys pack for the past 3 years…5 more years to go as my youngest is just a Tiger now) 🙂
Nice! The U.S. versions are a bit diff. http://www.usscouts.org/adv…
“I am an Eagle Scout”Impressive. Definitely one of those things that I could never have done when I was a kid! I remember seeing that book of the requirements. I’m more of an internally motivated person who hates forced structure and requirements. I dropped out of cross country in high school but have been running every day for about 15 years. I wonder what kids today think of things like that. I just remember growing up and being really impressed with anyone who made Eagle Scout because it was something that I wasn’t able to achieve.
i didn’t know that.i am [email protected] is an eagle scout as well.
i still have some of my skills
As I recall, it’s the Girl Scout motto too. A good one.
so sorry about your home, especially after fixing it after Irene. I saw the photo Gotham Gal posted the other day.
Please don’t feel badly for us. We are blessed. There are many who are not.
Ultimately, “the cloud” is still a series of big data rooms (many of which are in basements) that can just as easily be damaged. It’s highly likely that some backup systems were impacted by Sandy. It’s probable some files are irrecoverably lost. It always makes sense to backup important data, but we can’t 100% rely on anything to be foolproof. Best to do multiple backups across different systems if your digital records are critical. But there are no guarantees. And I don’t think there ever will be.
True. I obsess about this a little bit.
True. But I believe the big ones, like Amazon, have redundancy built in. Is that correct?
That’s how it has always been explained to me.
true, but if you are outsourcing your cloud to amazon, then you have to trust amazon (or whoever). there is always something to be justifiably paranoid about.
“there is always something to be justifiably paranoid about.”This is a ancient truth and the secret to many a fortune…
Sure, except most people completely underestimate the risks embedded in their own operations. Unless you are a 1 man business and your own system admin and are spending all your time keeping up on security, patches, etc, you have to rely on someone else, be it an employee, an IT consultant or so on.So all the paranoia about the ‘cloud’ appropriately *also* applies to your IT person as well, with the difference that Amazon has a multi-billion dollar business to lose if they institutionally start behaving badly with people’s data.
There is redundancy built in as far as network drives and backups.But often you are still picking a certain ‘zone’ and that equates to specific physical location that the set of drives are located in…if something happens to that one location, and you are not spread across locations via your own backups/plans…then you are still in trouble (this is why you see EC2 outages take down certain services and not others at times…because different zones hit issues from time to time).There is a cost in money, time, and maintenance to spreading across zones and being ready to swap between zones when needed…and so it’s still a balancing act of when, where, and what to do to have solid redundancy.
“There is a cost in money, time, and maintenance to spreading across zones and being ready to swap between zones when needed…and so it’s still a balancing act of when, where, and what to do to have solid redundancy.”Exactly. No doubt someone can and will spend countless hours getting this correct. Obviously any solution would involve multiple Amazon zones and another provider in the event that there was an amazon issue that would disrupt their structure. Depending of course on what you stand to loose and who you were. Maybe even more than that. I’m remembering that there was financial system redundancy backup located across from Manhattan which assumed that any issue that hits Manhattan would not happen across the river. Oops.You work down in the ship’s engine room so you understand all the things that can and will go wrong. But most of the people in management don’t even know the issues and have to depend on someone else’s perception of the risks in order to make a decision. I”m reminded of the guy at the amusement park who says the ride is so safe he let’s his own children ride it. And of course there are some parents that let their kids do motocross and other hazardous things. Doesn’t mean it’s safe practice.
ME, GRIMLOCK, HAVE DIRECT EXPERIENCE HERE.COUNT ON AMAZON CLOUD NOT SIMPLE AS SOUND. MUST BUILD IN OWN REDUNDANCY.
The size makes it a juicy visible target for an adversary to bring down or disrupt. (Ok that’s the “Mac’s aren’t a big target so that’s why no viruses” line of thinking but there is truth to that.)The complexity almost makes it certain that they won’t be able to think and address all the potential contingencies for failures of parts of the system. And the complexity also means that it takes so much knowledge to even set it up right on the customers end. After all, if they push the multiple zones in marketing to much, people will wonder why they should have to do that at all if the service is so reliable (and why should they actually if their architecture is so great?)One of the things about the Internet business is the single point of failure in so many things. After all, Walmart has their stores all over the country. So while a few were shut down on the East Coast because of Sandy, it will have a negligible effect on their earnings. Much different then the Amazon retail store going down 10 days before Christmas for, say, 2 days.
You need to build it redundancy for EC2. S3 is fairly redundant.
Thanks. Do you have redundancy of data storage even without redundancy of availability?
Depends on what you pay for/setup…default is no, but you can get there pretty easy with something like EBS
Those are the questions you have to ask:1. What is an acceptable worst case datacenter blows up downtime. 2. What is an acceptable loss of data.The answer is not zero. It can be close to zero but each 9 you get does two things:1. Cost a ton of money2. Introduce more opportunities for failure.That second is one that smart people don’t grasp. Put in automatic failover? What happens if that fails? (happens all the time with Cisco Routers….the really expensive ones, the master thinks its up the slave thinks its down, they fight and you go down) So it actually might be it you can reduce the amount of downtime if you require a human to turn on the failover. You would think a twin engine private plane is safer, it is not the facts show that the second engine takes you to the accident site versus dealing with the disaster to begin with.Its fairly easy to do one hour log file shipments (that’s after having redundant servers and mirroring data at the master data center), and go to several redundant centers, located diversely. Same for bringing those centers back up.The final question is how much do you rely on one provider? That is a very important one. Anybody that runs cloud services realizes that unless you are Google, Facebook, Intel, or Microsoft, you rely on somebody. The question is how much and to whom? So mirror across data centers and have automatic failover? You’re now dependent on one provider, their routers go down? You are down.Its fairly easy to do one hour log file shipments, and go to several redundant centers, located very diversely. Same for bringing those centers back up.
The big guys (Amazon, Google) have redundancy of pretty much everything. Availability of service, storage (storage, more so), etc. Most startups do not have the money or the time to build such extensive redundancy. Of course, you knew that already 🙂
Good point. The two are different.
Conversation with lawyer: “Yes, I have all your files. They are in storage and will take two to three weeks to retrieve.”Sometimes, you might as well not have the files in storage.Yes, the two notions are different.
Yes. One of their backups is in a datacenter we use in Delaware. The main one is in Irvine, CAAnybody that runs any sort of serious cloud services has multiple redundant backups. We use one in Louisville and another in Europe.I will say this. This is a reason its actually good to pay for services as then you can ask what the policy is. If it is free, they can say whatever.As to security. I will say this. Most people like physically having something but don’t think about the security of that. A good example is a credit card. My Dad said I don’t like using my credit card online. I said ok, you are afraid of having it go over the internet via SSL but have no fear of giving it to a $2/hr waitstaff where it goes in the back out of your sight.A laptop is easy to steal. A hardrive is easy to take. If is not your job and your company to provide redundancy then you won’t take it seriously. Everybody says they will but they won’t. I’ll give another example: I was negotiating with Disney and their lawyers had all sorts of questions/language about backup, security, and privacy. We spend a ton on each and even have a Lloyds of London policy covering this. It is our company if we screw it up. I got flustered because I knew where the data was that we were going to convert and told there lawyers “Do you know where that data is now? It is on a Server located under the receptionist desk. Give me a letter and I’ll have that data in 24hrs”
” It is our company if we screw it up. I got flustered because I knew where the data was that we were going to convert and told there lawyers “Do you know where that data is now? It is on a Server located under the receptionist desk. Give me a letter and I’ll have that data in 24hrs””Can you explain this further? What you wrote above is confusing.
I will say this first. This is a facility they have since sold. The critical customer data that we were going to host that had all the addresses and purchase history of all their customers was located on a computer under the receptionist’s desk. Under her desk, because if a customer had a question about a purchase she would look it up. I actually looked at the workflow and data as part of the sales process. I could wait and watch till she left her desk and simply physically take the computer.Yes you should back things up to a different service provider each night. No I don’t use Amazon because it is another level of abstraction that I don’t control. Yes you can get burned with a bad service provider.
” Under her desk, because if a customer had a question about a purchase she would look it up.”Exactly. Not to mention the cleaning crew at night coming in with vacuum cleaners and banging into it I’m sure. Or sucking up some loose cables (easily fixed but certain to cause downtime). Or installing a key logger. (By the way when .info went online years ago I was in their office. The root server sat right out in the office floor, serving up DNS, in a non-secure building that had easy access. Obviously that was fixed but it was like that for a long time (as were the Internet’s root servers at Universities for that matter).I had a case in my old office with the T1 where we had what was known as a CSU/DSU. I had a spare CSU/DSU sitting right next to the main one. Now when someone came into clean the office I was always there watching. One day, the cleaning lady put a damp rag right on top of the spare CSU/DSU over the air vents. And I was right there, 10 feet away keeping a hawk eye on what she was doing. And it still happened. Why was the spare CSU/DSU next to the working one? To long to get into the reason but suffice to say I had thought through why it needed to be right there.People tend to ignore the small details of why things happen in many cases or sometimes random unpredictable things happen.The billions we put into defense to make sure the oil flows? Great. But yet the “last mile” (the local service station) is not equipped with inexpensive emergency generators in the event that the power grid is cutoff. Nor are the street lights. The big thinkers and academic think tanks don’t ponder that stuff in advance. They don’t operate at that level. Although it’s obvious and easily correctable small details that can gum up any plan.
I believe in something I call “the kitchen table perspective.”From an economics standpoint, it means that when you’re going through a tough patch, you defer the payments that won’t hurt your credit, let the utility bill slide an extra two weeks so you gain the advantage of another paycheck, stow away a little cash when you’ve got it so that when you’re really down to the gritty edge, you’ve got that $100 bill in your innermost pocket of your wallet. You also have canned goods, but not so many that you’ve locked your cash into a non-liquid asset. :-)People who take the time to think things through from their grass-roots view might have the extra vision to think through your suggestions like generators for the pumps at the gas stations, but as you say, the ivory-tower sages are too far removed.I used to go caving in wild caves, so I spent some time thinking about disaster preparation from that perspective. There were two absolute rules: explore with at least 3 people, and always have three sources of light. I usually had a headlamp, a flashlight, and finally some candles and matches.I recommend three sources of light for my friends there in NYC and environs.
“But I believe the big ones, like Amazon, have redundancy built in.”Let me state emphatically that it is a big mistake to use the fact that Amazon has redundancy built in as a reason to not have a backup which you personally control of anything on Amazon. As a matter of fact Amazon has lost data, has gone down, and has had problems. They are a big machine with many points of failure. I would say this even if they hadn’t had any issues.As a simple step, for your protection of email for example you can forward a copy of all email to another free account or even two. Like to an identical account at outlook.com and also to an identical account at Yahoo.com email. That way if gmail is not working you can always access all your mail archived at those two services. (Of course you won’t get any new email but at least all your old email will be available.). This is free and it doesn’t take any effort once setup.
“like Amazon, have redundancy built in.”All these systems are designed by people. Look at what happened to the backup system at NYU and at Bellevue. And those were world class IT and systems design people most likely. Not the guy overseeing IT at the corner bodega.
Fake Grimlock has it right, with AWS you have to design in your own redundancy. It’s conceptually similar to running two of your own data center installations but instead of owning your own hardware, you are using Amazon’s services. But redundancy is not a built in feature of AWS. AWS is basically operating lower in the stack and providing good building blocks on which to build a redundant system. However, AWS will tell you where the data centers are physically located so that you can use data centers in different parts of the country.
They all have the capability of offering said redundancy. And internally they use it in their systems. But the onus is still on the application developer to use the systems in a redundant way.As someone that builds these sorts of systems for a living, I hate talking to folks that have moved only to go down when their piece of the cloud goes down. It takes using the cloud in a highly redundant way. Most don’t even know what questions to ask.I hope one day to be better at federating this sort of knowledge. I haven’t figured out how to do it yet.
It’s a challenging topic to blog about. It’s essentially giving architecture guidance in a blog format. Imagine trying to write a blog “how to build a website.” It can be done, it’s just hard. I don’t feel like I’ve come up with the right approach yet. Feels like it should be b-school case-study-ish…
that would be great too
depends. on what facility. and what product you buy from AWS. when one of amazon’s east coast facilities went down a little while ago this year, all sorts of sites and services (eg netflix) went down with it. a lot of folks discovered the hard way the gaps in AWS “redundancy”
Yes that is correct. When you are that big and trusted so heavily to protect data, you have to have some form of a backup.
Yes, that is correct — S3 keeps 3 copies of your files in 3 different data centers that are geographically distributed.”Amazon S3’s standard storage is:- Backed with the Amazon S3 Service Level Agreement.- Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.- Designed to sustain the concurrent loss of data in two facilities”While EC2 has had availability issues, losses on S3 (storage) are exceedingly rare.Sure, all the comments below are true that no human-designed system is absolutely perfect, but the reality is that S3 is a couple of orders of magnitude more capable than pretty much anything a company that doesn’t have a 9 figure IT budget is going to put together, let alone a family, is going to do on their own.If one reads in detail Amazon’s High Availability processes and procedures, it is everything you will think of on your own (distributed data centers, backup generators, 24/7 coverage with engineers, etc, etc), plus all the things that you haven’t thought of because you haven’t been running one of the world’s 5 largest data stores over the last decade. [note, I have no association with Amazon, except I am happy user of AWS for personal and business purposes]Here is their post-mortem on their EC2 outage over the summer to give you a sense of what is happening (and this is a storm that took out Verizon and 911 service during that period for several days):http://aws.amazon.com/messa…If you are storing on S3 + keeping a local copy for redundancy the odds that all three Amazon copies and your local copy will disappear simultaneously are infinitesimally small.For this to happen, you basically have to assume a successful, malicious hacking attempt targeted at you that manages to get access to both your home PC (or your safe with an archived external HDD) and your cloud credentials as well at the same time.And, should that happen, well, you probably have a bigger issue…
‘cloud’ seems deeply ironic
Several data centers in NYC went down or were severely strained over the past week. Servers were not in basements but fuel tanks for diesel generators (typically on rooftops) and fuel pumps – to get the fuel to the roof – were. At 111 8th avenue, several generators blipped or were operating below full capacity for several hours.
I saw your previous comment and I know there are many who are not as blessed, so I won’t “feel bad” for you, but I still don’t like to see anyone go through this stuff. I’m glad you have the resources to recover from it, but I know the next couple months will be challenging and/or inconvenient. I also know you will probably learn a lot from it. And share it with us. 🙂
Two weeks back. 7.30pm Saturday night. 5yr old daughter bounds down the stairs holding my uncased iPad. Before I had a chance to scream “STOOOOP” – she slips, iPad flies out her hands spins in mid-air and slams screen down onto our tiled floor.Slowly and calmly, whilst telling her I wasn’t mad with her (I was SO frikking mad with her), I walk over and pick it up. Screen was smashed to a gazillion pieces. – ARGHHH.I left the house, zoomed to Apple, got to the store 10mins before it closed. They swapped it for a new one without any stress.I get home, turn it on, enter only my Apple and Google App details. Within minutes everything was restored. Email, contacts, calendar, apps, photos etc.60 min round trip from having a dead iPad to a fully restored brand new one.Cloud rocks.
This is a massive improvement to where we were with setup/config only a few years ago
Me too. My iPad was wiped clean by a bug in iTunes while upgrading from iOS 4 to 5. In the end, my “stuff” was restored, not from the iTunes backup that was made just before the OS upgrade but from the Apple store which had a full record of the apps I had installed.
My question is why do you have an uncased iPad? You should know that the screen was glass, and can and will smash into a million pieces. This is why I prefer gorilla glass over normal glass because gorilla glass is much harder to break.
I thought the iPad was made wtih gorilla glass – if not then which tablets are?
I don’t know.
The apple extending warranty always comes through in these types of cases
Very happy to hear that you’re able to weather this disaster better than so many others. Still a nightmare, though and I don’t envy you the next 2 months or so. It would be great to know how you rebuild your home network (set-up, equipment, etc) and what cloud services you use for what.
I’m curious about how you handle the remaning official documents and paperworks, like deeds, certificates of ownerships, etc..all types of official or government docs. Do you have them in a fire proof safe at home, outside bank safe or other?
File cabinet in my home office which is not fire proof but is on the top floor
Is there a service that could take these docs, certify scanned copies, and store them in the cloud?
I bet there is
When I looked a year ago there were no services.
It might be a good business. There are services like that for corporations I think, but perhaps not to the consumers. Even banks were more entrepreneurial, they could start something like that, because there’s a trust factor already in place.
For small business owners there really should be something. I wonder who will jump in to do it. Trust factor is huge, which is why, in my comment above, I just did it myself.
Certification of scans would likely be incredibly high touch.
“There are services like that for corporations I think”Most definitely. And that is where the opportunity is.When confronted with an opportunity like that and a choice to sell to either corporations or people choose corporations. Businesses pay money, people do not. (Unless you can make it up in volume and this isn’t one of those opportunities if you are talking about doing the scanning).Think about it for a second. You have to get the documents (to your facility which involves shipping which is weight sensitive) and then you are dealing with someone spending their own money instead of someone elses. And you are running up against the “painters” theory that I have. Which says that anything someone can do themselves fairly easily (painting vs. carpentry for example) they are more likely to do a shitty job themselves if they have the time available. And in the case of scanning things yourself using snapscan is an option.So really what you are talking about is having a service whereby it’s easy for people to take their existing scans and have them accessible in the cloud. And in fact there are ways to do that now. I’m not sure to what extent its a hack (like using dropbox) or some simple thing that can be integrated and made easy for people to use. But if there isn’t the exact thing out there there is something very close that is available to use.I do know that there was a service that I read about a few years ago that handles scanning and putting into the cloud’s people’s documents. To me the pricing seemed to expensive for individuals (and it was targeted towards people and business). But I can’t remember the name and a quick search didn’t find anything.
I was looking into setting something like that up. I purchased several high volume scanners in order to digitize about 50 boxes of paper that are now in storage. It became apparent that it would be a decent business to be in. I checked into the rates people charge for scanning and did some preliminary research and time testing. The big bottleneck is not the speed of the machines. The machines are fast. The issue really is getting the documents broken down for scanning. People’s and businesses documents are stapled, in file folders, in notes taped onto file folders. From the tests I have done it takes about a 30 minutes to an hour to prep 1000 documents. Those 1000 documents would only take perhaps 10 minutes to scan. So that is the bottleneck. And to cure the bottleneck you need labor. After the documents are scanned they need to be destroyed. That’s also a great business. Much much less to fuck up. Hard to mess up shredding. Mobile shredding trucks are an entire industry. (You will see them in Manhattan all over.)There is also the issue of getting people who do accurate work operating the equipment. I know much about this because I owned high speed copiers (legal copying) for years and getting people who do a good job is not trivial. And you have to constantly check and make sure they aren’t missing documents.Scanning: Even in the testing that I did I noticed that when you OCR documents using, say Adobe acrobat (which is not the best solution I’m just mentioning a “gotcha”) that it actually truncates some pages. A bug. And it happens. So if you have others in charge of the quality control on this you can run into all sorts of issues down the road whereby your clients are missing info. Not to mention a host of other things that can go wrong.All in all I think it’s a great business to be in. The cloud part is easy. It’s the part of getting it into the cloud. that requires some thought. That makes it a good business with a nice barrier to entry against competition.Here is a “moderate” speed scanner that I sold that I didn’t need a few years back. A woman in Miami bought it and it was exported to south america. The video that I put together was helpful in getting the top dollar for the sale. We were able to drive it with a cheap PC and Windows XP.https://www.youtube.com/wat…
Companies doing medical records or legal records archiving have the technology for high-speed/automated archiving. I think it’s a combination of off-shore outsourced services and software.
there are fireproof file cabinets?
I have one to but I look at it differently after my brother in laws parents house was completely burned, and we had the job of sifting through the ashes. Nothing in the fireproof file cabinet on the top floor was saved. Everything in the plastic totes in the basement was fine.Now you certainly can’t draw a line from this point. They were in the middle of a five house fire in an old town and the firefighters correctly sacrificed the middle houses to stop the blaze from spreading downwind.
Big expensive and really heavy. I had several in the 80’s. The logistics of getting one to the top floor of Fred’s place would be interesting. They are also slightly harder to get into than a typical file cabinet which practically any thief could gain access to. The mechanisms are more secure.
Our house in CGY (which we built) had a ‘cold storage’ room, which is basically a room with foundation walls & a concrete slab on the top. I used it for lots of things, but it was great for vital doc storage.Not sure what I am going to do in our new house in Ontario, but fireproof file cabinets sound like a fun thing to spend time researching.
I think so…Is the cold storage room used for wine too 🙂
Yes, but I have tannin allergies, so we are white only& don’t lay wine down.
I was talking to a friend of mine who is the head of digital production at a big agency in Toronto — he was saying that his company spent hundreds of thousands of dollars on a part of the infrastructure (servers in building, proprietary system etc.) that he would have replaced with google for basically free.
I like that you used the word decommission in reference to objects in your home. I’m stealing that, and, later today, I’m going to decommission a few old towels and yesterday’s copy of The New York Times.
It is a fun word and a fun practice. The Gotham Gal is the queen of decommissioning
Just curious – did you use a service to transfer old photos to digital? If so, which one? I’ve been thinking of taking the plunge and digitizing 40 years of photos and getting them to the cloud.Honestly, they are about the only physical possession I would miss.
I have not. We still have photos in albums and shoe boxes (actually they are in big plastic boxes from the container store)However I think doing so is super smart and something we should do
Mike, ‘digitizing’ is gruesome.Photos are one thing (yikes).I have scanner related carpal tunnel issues (kidding), trying to keep our family records in the cloud.
I have done it for 1000s of negatives and slides – shttp://www.scancafe.com/ . They did a great job.
I second this comment. Scancafe is amazing.
“Moving to the cloud” sounds complicated until you do it, after which you realize that in some ways it’s much less complicated than doing everything locally. The cloud not only gives you disaster resiliency, it simplifies your computing life overall.I’m already there but I didn’t know that a dedicated Quicken user would be happy with Quickbooks Online — that’s good to know & my next project.
Also seeing more clients using Freshbooks as an alternative. If you don’t have inventory or payroll, it is worth looking into. For personal finance I dropped Quicken and went to Mint.com – now an intuit company.
QuickBooks online is not as good as Quicken. It is designed for business and is missing a lot of necessary features for personal use. But it has one that trumps all else which is its in the cloud.The fact there is no Quicken Online is a complete and total embarrassment.
I liked Quicken better than QuickBooks, which I detested. I really wish they would beef up Mint.
When I see something as obvious as that (clearly the need) there is probably some angle that is not being thought of that is the reason. Maybe some patent issue, maybe some security issue, maybe some agreement, maybe some other planned strategy for another product that is delayed that will serve that need. Could be also a simple as not wanting customers to get used to doing that in the cloud so it is easy for them to switch to another cloud company. Remember now they can sell quicken at a low price for basically a disk or a download in an existing channel (could be a channel issue for that matter). If they go to the cloud they have to operate a entire infrastructure (with security issues) which is way more expensive then simply download and disks. And deal with a different set of cs issues.
Intuit (which owns Quicken) bought Mint.com (good online personal finance site), and I think there might eventually be some compatibility between Quicken and Mint. The sooner the better b/c Quicken is getting very outdated imho.
Mint needs to be able to make reports and saved P&L etc which Quicken does. I love mint but now cobble together this info which would be singe click in Quicken and QuickBooks if they’d give you the feature to set up. I have not understood why they haven’t done this yet – it’s so obvious.
I would try finding out about redundancy for services that are not amazon or google.If only because I know my dad is using rad2, but it is in the basement….
I started scanning all my documents a while back (well, my assistant did the grunt work). Thick tax returns (due to long term former Real Estate work), and now I’m on to sorting out which closing binders and other docs are needed as back up. I detested that RE work which took so many years, so having all the space tied up with paper was psychically draining. I was dancing a jig to quit that job and move to my start up and my passion full time.Back when I decided to scan and purge, I asked around to my CPA, legal, and financial friends for a secure scanning service and there was none they knew of, nor any we could find. So I made my own makeshift system and started scanning and then shredding. Many people looked at me like I was somehow possessed, or OCD. I am now comforted that others like Fred, have moved to paperless/cloud.Figuring out exactly what was needed to keep, in my case for IRS, is a task unto itself when you have boxes of paper. I have only recently become the ‘new me’ with Dropbox, which I love deeply for how easy it has made my life. I still have drive and DVD back up just in case.I wonder if it would be a service to anyone interested in going to cloud to list some of the providers who might do this work in bulk as William mentions in this thread. For documents, photos and any other types of things people may have. I am also scanning medical records.I would love to see services where people can have their vital documents, perhaps even providing lists for people of what they should have or might need. Financial and legal is one segment, family photos and memorabilia is another, and medical would be amazing. I wish the medical could also provide graphing of some of the data too – like blood work history for example, and family histories. I really feel like it would be so helpful for people to have control of their data, as well as the obvious benefit of disaster prevention of loss which sadly many people have now due to Sandy.
I’m curious what service you use to store your photos online? This is something that I’m looking to do as well – and I haven’t found a great solution.
I use generic cloud file storage but I hear good things about this one https://picturelife.com
When you’ve operated a digital business in Saigon, you learn quickly to not put anything sensitive below the 2nd floor. And if the business is serious, you put $20k of generators on the roof, soundproofed for the neighbors’ sake (further raising the cost). Because floods and brownouts/blackouts are routine there but you can be close to impervious with the right planning and investment.When you’ve lived in California, you acquire a habit of never putting glassware on top of cabinets or close to the front of shelves. And you still do that even after you’ve moved to a non-earthquake prone place.
where do they put things like boilers, hot water heaters, pumps, and electrical panels in Saigon?
Can only say what *we* do at our 3rd and newest HQ, having learned a lot:We keep our 100kva diesel generator behind a gate and under a canopy at the front of our building. It turns on automatically, but every computer has its own UPS to keep power running during the several seconds it takes to start up.We have > 12 A/C units mounted along the back wall of our building (staff size now 50 and growing).Our main electric panels are in our basement and at the front of our building with separate ones on each floor as well. A water pump for drainage is in our basement, and a pump to move water around the building is on the roof, along with our water tank.Fiber optic Internet cables enter on the 3rd floor, where we also have a rack of network switches. We haven’t had a down day in over 3 years, despite typical years laiden with floods and outages.Flooding in Saigon is so common that it doesn’t get much attention. The buses still run; children still wade through the water on their way to school. Some of the worst flooding is smack in the center of town. Our last two offices have been in locations that are high enough to never flood. Our 1st in the hip center sure did. See http://www.youtube.com/watc… near our first office.
NYC is going to have to learn from cities like Saigon
the biggest lesson is simply attitude– most 3rd world countries rely on shrugs and laughs. the video i linked to shows this.
I wonder how using fiber optic internet is. It must be crazy fast — as well as expensive.
relatively new offering in Vietnam, and available in the area of our new office. we tend to sign up for a mixture of whatever connectivity we can load balance together, this is yet another option, not too much more costly than the other conduits we use.
ME, GRIMLOCK, APPROVE IDEA OF EVERY BUILDING BE OWN SELF-SUFFICIENT WALLED COMPOUND.NO FORGET TO INSTALL TURRETS AND MACHINE GUNS.
And sharks in the moats? Didn’t I see a picture of that in the storm photos?
I was without power or cell/internet service most of the week (we are back now but most of my area out in central NJ is still without actually)…and luckily we had no flooding or real damage (a few down trees that I’ve got to take the chainsaw to).At it’s current stage gawk.it is still a small one-man show and so throughout this week I had no real way to monitor the status or stability of the service…I simply had to trust and hope that it remained steady and stable in the cloud throughout (happily it did)…so I was lucky, but it def. has me thinking a lot more about systems and procedures for ensuring long term stability regardless of my own life status/situation…
Hey Kevin aren’t you actually in North Jersey? To people in South Jersey, Princeton is central anything up toward the city is North Jersey.
I live in long hill township (1 mile north of 78 and about 40 miles from PA border)…could be north NJ to some I guess…I think of it as central in terms of NY vs PA (east/west) not so much north/south.
We have been hacking “cloud” systems such as email and document systems to store the family information jewels for a while now. Makes life so much easier even before disaster recovery.Something has unsettled me though.In many ways, we are letting our most dear info rush out into the information super highway hoping it does not get flattened. We need to clarify the following rules of the road among others:Data ownership (clarify who owns the personal info in Google or Amazon’s cloud;Data stewardship (the cloud provider’s baseline responsibility to protect from third parties and snoopy employees)Data morticians (how disposal is handled when company dies or sold);Data portability (my rights to move data elsewhere);Right to be forgotten (my rights to have my data completely overwritten);Fourth amendment reasonable expectation of privacy (protection against search without warrant)
All very good points. A lot of this is very well figured-out in old-fashioned safe deposit boxes: e.g., the contracts allow for what happens if the bank is sold, and also I know that the boxes can only be opened with a warrant — the law recognizes them as personal in the way your home is, even though it’s not your home but bank property. And of course portability is a given, since you’re moving tangible items anyway.I’m not suggesting safe deposit boxes are the future! just that a lot of the issues and questions are not new, and there’s a helpful precedent.
INHERITABLE DIGITAL SAFE DEPOSIT BOX WITH BUILT-IN TRIGGERABLE RIGHTS TRANSFER?THIS GOOD STARTUP IDEA.
I agree. Build it to store Bitcoin too
See the “Security” paragraph here… talking about “digital bank” (the post is 7-years old):http://sotirov.com/2005/02/…
I like it!
from my experience in the financial world, there is a viewpoint that if you are not paying a custodian fee and you don’t possess it, you don’t own it — and so many people have skepticism towards banks and brokerages that dont charge a storage fee (late fees do not count). emails gotta be worse.i’ve slacked off on my paranoid game. i need to get back on track here. anyone have any thoughts on hushmail?
And leaving your online data to a family member in your will when you pass away.
Nice catch there. Digital I would assume is not so well willed or managed.
It’s sort of a black hole but I figure we’ll figure it out before 170 years from now when I expect to pass on. 😉
Nice one!Do pass on the secret to the extra 100+ years!
Oh, I’m just planning on that getting invented during the next 30-60 years. 😉
There are actually a few startups tacking a stab at this, e.g., Legacy Locker.
Interesting. I’ll have to take a look. I was thinking it might take laws to treat digital property like tangible property, like the one Fred is talking about supporting. Imagine that…a law written by Senator Leahy that I could enthusiastically get behind! 😉
Related NYT Magazine article from last year, Cyberspace When You’re Dead.
This is very insightful. I am big a cloud advocate, but few people give as much thought as you have about moving everything to the cloud. A lot of our work at ShuttleCloud has been around cloud data ownership, portability and stewardship. I’d love to hear more about your thoughts on this topic… ping me on Twitter (edufernandez) or email ([email protected])? Thanks!
also, the cloud is in the dirt somewhere …
Well said. Serious concerns.
Data stewardship is a big one. Several years ago, I did some freelance work for a Big-4 accounting firm. One of the firm’s employees lost a laptop containing the tax returns (including Social Security numbers and other personal identity info) of millions of customers. Why on earth should anyone have been allowed to download all that info to a laptop in the first place? Made no sense.
i would need to see a more rigorous economic analysis to have a lot of conviction, but i lean towards wishing the nyc marathon went on. not because i care about some stupid race but because it comes down to money. money for the city means money for fixing things. i’m glad the knicks played (and won!). life needs to go on after disasters, if it didnt nothing would happen because there are literally disasters going on all the time, the majority of which are way bigger than sandy.
“i care about some stupid race”I don’t care about “the stupid race” either. But you are a sports fan so I’m surprised you would think that way. Or call it stupid.”money for the city means money for fixing things.”Hey, the people are already there spending money! (But read below more).To me you have to separate the emotional issues “people are suffering so don’t run the marathon” from the real issues and reason to cancel. Resources.I don’t believe it is a reason to cancel anything because there are people starving in Africa. Does it make sense that Fred is going to take his money to rebuild his media room and lavish lifestyle when people are homeless in Staten Island? It doesn’t matter. It’s not up to anybody to not do something because the money could be used elsewhere for greater benefit. So the “way things look” and “respect” have nothing to do with it.Why? Because all that matters is the fact that cancelling this will have economic impact on someone. And it’s not right for that someone to have economic impact (like the tshirt vendors who stocked up or the others who spent money and prepared for this and traveled here) just because there will be those with hurt feelings or feel it’s not appropriate. (Or don’t have economic impact and can just pile on to the lemming sentiment and then butter their bagel.)Nobody is compensating those people who will loose from the cancellation. And I’m sure that nobody who is on the sidelines (who doesn’t care about the race I certainly don’t) is offering to compensate any of the people who will loose out. So it’s easy for them to talk all sorts of smack about the right thing to do. Because they have no skin in the game.But there is one issue that does matter. The resources of the city that have to be diverted in order to hold the race. That trumps everything. That is certainly a reason to not hold the race. The other reasons aren’t.People are always forcing other people to do shit when it doesn’t require them to loose anything or it’s something they don’t care about. So while I don’t care about the race I also recognize that I have nothing to loose at all by it not running.
I am with you on this one. I am a big fan of getting back to normal whenever possible. I think Bloomberg and the Marathon folks blew a bunch of things. I was hoping they would do it bare bones but do it.
For me, it was a matter of documents and space. In the first four years of my service on the community college board, I filled a four drawer filing cabinet. Since then, I’ve put every single paper document in the cloud and I keep almost none of them.For documents, there has been nothing better than the Fujitsu ScanSnap (fast, double sided scanning) and Evernote Pro (1GB a month and OCR indexed for search).
NOW JUST NEED MORE REDUNDANT CLOUD.
in the cloud
What about security and privacy in the cloud? My alma mater has moved to gmail. I can easily see an engineer who wants to get their child admitted peeking at the traffic. I’m sure you have a lot of sensitive business communications in your email that might impact a deal. For example, if Google is in negotiations to purchase a portfolio company.
I don’t worry about that. One slipup like that and they will lose millions of customers immediately. They are in the trust business and they know it.
I do in fact use gnu privacy guard (gpg). The key management hassles mean that, except for some serious open source developers, all of the people I email with don’t use it.
Make me smile! Phil Z was a cubicle buddy of mine in the ’80s. A fan of Garrison Keillor, as you might know, he was warm, funny, and brilliant. Full of personal integrity, I’m glad to see his impact on personal privacy on the net.
When we started our new business, a service business which depends on real-time data to supply customers with energy, we started with a “no physical server” rule; even the “switchboard” is in the cloud. QuickBooks online, of course, even though it is missing some of the features of the desktop version.We aren’t in service yet so haven’t been fully tested; but having employees work from anywhere and everywhere and being able to move through a couple of temporary HQs without ever being “down” or worried about moving servers has been invaluable. I’m sure we will be tested eventually by a storm or some other catastrophe; we’ll have to worry about getting our trucks through. But we won’t worry about our data or our computing and dispatching ability; it’ll be elsewhere.more at http://blog.tomevslin.com/2…When I was volunteer Vermont CTO I tried, unsuccessfully, to move State applications to the cloud. “how can we know that Amazon is secure?” was a successful counter argument.The Irene happened; many of the servers ended up underwater. Sure, there was backup data offsite but it was weeks before some critical systems were running again. And many of them were the systems you most want access to in an emergency.Even if you’re data center is as secure and redundant as Amazon’s (it probably isn’t) there is good reason for having your data and applications a storm’s width away from you. You want to minimize the possibility of a virtual; and physical meltdown at the same time.more at http://blog.tomevslin.com/2…
coming from the guy who brought us exchange server!!!!
Which now has a cloud option… although I think the best cloud services are designed for the cloud and not evaporated upwards from the company data center.
This is why I like to use 1 server with monstrous parts. If you can travel to the server location, shut it down and move it somewhere, then you will have no physical damage, but then networks would go down.
Hi Fred. Similar to yourself, i am a bit obsessive about tracking personal financial records as well. I still use Microsoft Money (they have stopped supporting but i’ve been using since college). I had thought the quicken cloud service (at least when i tested it a year or so ago was still pretty weak and inadequate relative to the desktop version) so I decided to store my MS money file in the cloud (using dropbox in my case) and have Money installed on multiple computers running windows and so have a bastardized version of ‘Money in the cloud’. Works just like storing a Word or Excel document in the cloud. Anyway, I don’t know if this would work as well for Quicken but this has been lifesaver for me.
I post the following NOT for the political sentiments, but for the observations about computer security that are right on topic for today.http://news.yahoo.com/obama…
Just read this post *moments* before getting a tweet about an enterprising soul in Peter Cooper / Stuy Town that thought enough to put up a Google Doc Spreadsheet with each building address and the status of utilities (water, heat, etc) at each – then blasted it out for everyone to update and keep accurate. Sooo much better then anything that property management has done. I shared the doc with my wife – via FB – who has a ton of “mom friends” across the community….this thing will be in 50 more families hands within the hour and a thousand more the hour after that. #cloud #socialThe doc is public so if there are any Stuy Town / Peter Cooper residents on avc.com – here you go.https://docs.google.com/spr…
For the first time in my life, I witnessed a hard drive fail last year. It was a family pc and had over 15 years of Quicken data on it. Had to start all over from scratch and we use a combination of flash drives and cloud backup (not Dropbox b/c of their security issues). So Quickbooks online resonates with me and I might migrate there eventually. Also use Mint which is great as well. I never took online backup seriously b/c of security concerns but at a certain point I guess you have to let it go. Call me crazy but I’m trusting Google less and less with my data and instead going with Microsoft since they’re not constantly mining data for ads.
i worked with a shop that literally had a server crash — the shelving holding the servers all came crashing down. no redundant backup. most stuff got recovered but they were not nearly as smart as they should have been; reminded me of when people in lower manhattan discovered after 911 that a backup disc in a drawer in the same bldg as the big machine is no safety at all
I also live in Manhattan and while not a full cloud convert, do believe in off-site backup. Last year I stored my backup files in Colorado using a service that lets customers chose from a handful of physical locations. I chose the company, in part, because of the location transparency they offered. This year, I have added a second offsite storage location in Europe. Each is used for what I consider my most important files which, after being zipped and encrypted, comes to only a few hundred megabytes.I also store files in a local safe deposit box, but it’s a bit of a pain to access. That said, since the bank is well above sea level, its a reasonable option for large amounts of data. Wasn’t considering sea level issues when I chose the bank. In the back of my mind though will always be the images of the ruined safe deposit boxes that were under the WTC on 9/11.
Here’s my flood story, complete with me carrying a fibre channel switch to the boat.. http://dmevolve.com/2010/03…My heart goes out to you and everyone else affected. It’s a scary time. While keeping the apes safe, several of our staff contracted infections from the flood water.
if that company is the one I’m thinking it is ..I was quite surprised they didn’t already have their stuff in the cloud. In any case .. I think they “failed gracefully” and got their shit together pretty quickly – kudos to them!
I just posted a reply to Fred elsewhere here, but wanted to include this clip 1 more time separately as it shows a typical flooding day in the streets of Saigon. Notice the mostly scooter traffic plowing right through the waters. This was in front of our office when we set up shop in 2007:http://www.youtube.com/watc…
What do you use to store all your family pictures? I haven’t found really cheap storage for that anywhere that’s reliable. I won’t likely access it often. I just want a place to store (even backup style) my family pictures. Dropbox, but 500Gig would work. Thoughts?
I’m trying a new one that launched this week called MyShoeBox http://MyShoeBox.com I have also been using PictureLife
I use the cloud for everything. SkyDrive and the rest of the Windows Live suite of services has never done me harm, so I’m sticking with them. Windows Live is my collaboration suite for my techmansworld.com domain.
The data on my and my wife’s computers (laptops) is all in a few main folders:1. Dropbox (for all kinds of mostly personal files)2. SVN (for stuff from my business)3. GitHub (for all development)4. And we use G-Drive of course.The Dropbox folder is backed up through Mozy.All our email boxes go through a few services – our Bluehost hosting account, Gmail, and again Gmail (shadow accounts where the email is only stored as a backup), and G Apps for our business emails.I can stop typing something… on one of my laptops… and continue typing on another (thanks to Skype, Gdocs, and Gmail). I can do the same with a Photoshop file (thanks to Dropbox).My business’ sites are on AWS (but of course).Love the cloud(s)!
Fred – this post resonates with me as I’m trying to do the same thing. We just bought a Pogoplug as we realized our college aged daughter can’t get access to our shared network drive from campus – plus the idea of archiving 100 GB of data into the cloud (or more) makes our pictures/videos/music accessible anywhere.This may seem like a goofy question, but I too have been using Quicken for many years to track finances, and I’ve used Quickbooks Online for one of my businesses – how did you find the migration of your personal finances to QBO? Were you able to migrate all your Quicken data (I have mine in several archive files)? Isn’t it overkill for personal finances? This is the last main piece of software that I’m tied to my desktop, and it would be good to untether.On a final note, we are still out of power here in Central NJ and our hearts go out to everyone affected by the storm. Pretty humbling to realize what life is like without our basic modern conveniences.
Born in Rahway, lived in Bayonne, Newark and Bridgeton. Moved out west, but still have heart connections to you folks “back East.”Out of power for a week or more in Arkansas after a storm system that kept NOAA busy with pictures for a month, I learned that the most important things were to keep the laptop and cellphone charged, and keep an eye out for a shower. Truck stops seem to continue to offer them, although the prices are high. Of course after 8 days …Unfortunately there wasn’t enough white lightning to cover the human odors. Humbling, yes. Remember that what you feel like right now is not “who you are.” You’ll have a way to recover soon. Do whatever you can to comfort yourself and celebrate your life and those who you love around you.
thanks Wavelengths…. we know we’ll get through this. We’re much better off than many, so no complaints here…. but we all have a strong desire to return to “normal”. Patience is key, and lots of empathy…
you have to do a migration from quicken to quickbooks then you can go quickbooks to quickbooks online. you lose some details on the migration from quicken to quickbooks but the key data does come across
oh…. don’t really like that option…. I’ll keep looking for something else while I continue to use Quicken. Thanks….
I’m glad you wrote this post. A lot to think about. What I know is that most areas of business are not where people in tech field are, not even the big guys. My place of previous employment, which shall remain nameless for the time being, until it hits the news, lost thousands of hard copy documents due to the horrendous flooding downtown. A lot of it may have made it’s way onto one of the poorly organized and outdated electronic databases, some of it might be recoverable by email, but much of it won’t be, and it’s unclear how they’ll rally. Many of the old folks on top still haven’t adjusted to the times. An important conversation that I hope continues both on a personal and corporate level.
Shouldn’t be that hard for Dropbox’s of the world to offer encrypted storage options….
as Dropbox does encrypt data (AES-256 bit encryption) is redundant to upload encrypted data, but depending on what it…hard to be too careful.
We built a series of applications to manage various aspects of the home, primarily centered around home inventory for insurance purposes, but also home maintenance and home improvement projects. Stats say that 70% of home owners do not have a home inventory.John BodrozicCo-Founder and CEO of HomeZada
Too bad Gawker, Huffington Post, etc didn’t follow your model!
Yep, amazon gives you the tools and a platform that provides redundancy. EBS has redundancy built in. Certainly a long way ahead of running your own small datacenter in a basement. S3 takes it a step further. The ultimate step would be to use multiple cloud providers. In more depth – AirBNB & Reddit didn’t have to fail:http://www.iheavy.com/2012/…
That is messed up and wrong. We need to insure that the Fourth Amendment extends go our online assets. Sen Pat Leahy has a bill that would specifically provide for that protection. We all need to support that legislation.
Where is the logic in that? My first thought is to wonder how this same logic applies to funds that are not in our physical possession?
WHat’s the best way to support that bill?
Possession as the saying goes is “9/10ths” of the law. If you are not in physical possession of something it opens up someone else as the gatekeeper of that thing. And that person or entity can rightly or wrongly give up that information or physical object. Make a mistake or deliberately. For any number of reasons. Think that google or any startup is always going to go to the mat for you since that might be their stated policy and what they have done in the past? Well remember what happened with CBS and the tobacco companies (the movie “The Insider” where CBS caved to their principles and didn’t run the story.. Ask Mike Wallace about that one and why he did what he did. In his own self interest (according to the movie version of course.)) Things change. There was that one NYT reporter that went to jail for a long time because she didn’t want to reveal her source (Judith Miller). My guess is that this would be atypical behavior when faced with prison time.Let’s take two cases. You have a physical object in your house. So the government has certain steps to follow to get access to that object. Take another case. Your friend has possession of that object. Or a storage facility has possession. Each is a different barrier to cross for someone wanting that information.I don’t find it surprising at all that the government is making this play.By the way once that info is in the cloud it lives on even after you delete it. On other disks, in backups in other places. Etc.
It will never pass.I’m sorry if I sound like a tinfoil hat wearing conspiracy theory nutjob, but the evidence is overwhelmingly in favor of the US Government actually preferring to have this backdoor access to everyone’s data.
Online advocacy combined with old fashioned lobbying. We are doing both.
Oh dear. You wouldn’t actually be suggesting that we couldn’t TRUST that everything will be OK?I’d bet someone has already thought that through and is hoping the rest of us don’t catch on.