It is tempting to search for the one magic move that will make everything better. A new VP of Sales. A new database layer in your tech stack. A new brand for your company. Moving everything to the cloud. More capital in the business.
But it is rarely one thing that a business needs to succeed. It is often a little bit of everything.
Back in the early days of Twitter, we could not keep the website and API up. We would hire advisors and they would recommend something new and we would try it and we would still go down. It was terribly frustrating and threatened the business.
During this period of instability, Twitter purchased a search engine called Summize. Summize was a small team of engineers, most of whom had come out of AOL.
After we cut the deal to acquire Summize, I asked Jack Dorsey, who was running Twitter at the time, how we planned to integrate the Summize team. He looked at me and said “we are not going to integrate them, they are going to integrate us.” And Jack made Greg Pass, Summize’s engineering leader, Twitter’s engineering leader.
It was interesting to watch Greg and the Summize team tackle the “fail whale.” Instead of searching for a magic solution, they instrumented the entire system and just started rebuilding every part that was about to break.
It was a slow and steady approach. It took time. But within six months (or thereabouts), we had a much more stable system. And after about a year of this approach, we had mostly said goodbye to the fail whale.
Grinding isn’t very satisfying. It is hard to stand up in front of everyone and say “we are going to fix things around here bit by bit with a lot of hard work.” Big flashy moves are an easier sell most of the time. But they don’t work nearly as well and are prone to complete and abject failure.
If given a choice between a flashy operator or a grinder, I will take a grinder every time. It is a much higher percentage bet. It requires faith and patience and the results are sometimes hard to see. But if you look at the results from grinding it out over a long enough time frame, you can see the power of that approach.
Truer words were never said.Spent a big chunk of my career as the fixer, hired gun brought in to public companies to turn them around brand and distribution wise, startups to figure go to market out.The largest wins with some crazy hold-on-for-dear-lilfe exceptions are always a grind.
I like to waste time in a better design upfront instead of grinding later but experience has shown me that many times it is a waste of time instead of an good investment.The main reason is that it is more likely that flashes get financed, research and development does not. We have to ask a VC why this works this way.Flash, then grind.
Flind. Hat tip to @jason wright
Flind all the way, bro. 🙂
I hope you’re fline with that.
Thanks for this Fred. It’s good to be reminded of the exponential power of linear improvement 🙂
The quantum mechanics of talent. The Grasher.
I feel seen. On the software engineering side some credit to this sort of approach should go to Martin Fowler who coined it the “strangler pattern” (https://martinfowler.com/bl…. It is quite effective.Besides instrumenting the systems generally one of the biggest advances in software engineering is the practice of automated testing. Once you have identified the failing cases and edges of a system you can document these cases in code that checks the system for them. This makes it easier to see progress as you grind.When facing an unknown system many new engineers might want to do a re-write. Unless the system is small this is almost always a bad choice. One of the most challenging projects when I first started was when we decided to rewrite an old system. Spent down a large chunk of the budget replicating undocumented functionality. However this same principle won me my favorite victory when the City of Boston attempted to rewrite a thing I made and abandoned the rewrite In favor of just updating the thing I wrote.
Your last paragraph sums it up basically. I would emphasize 2 points:1/ the grinding needs to lead somewhere within a reasonable timeframe, otherwise I have seen teams grinding “forever” going in circles [forever is 3-4 years in startups].2/ OK to warn on flashy operators, but too much grinding without some flash is also not desirable, as your work could go undetected or under-appreciated otherwise.
There is no limit to the number of ways to mess up. Usually good advice outlines the main path of success, but it has to be implicit that we don’t mess up along the way for just nonsense reasons. E.g,, I have a nice recipe for pizza, but if just can’t measure the flour accurately the recipe won’t work — have to assume implicitly that actually can measure out 1 kg of flour and not take 3-4 years to do that.
+1 and I also think having founders/leadership grow their capacity to articulate and drive roadmap for experiments vs. core operational excellence goes a long way. The answer is really to do both, but not having a clear, company-wide understanding of where risks are acceptable often leads to eroding an existing customer / community base.
It takes a lot of guts for an Engineer to admit what they and their team have built needs a totally new set of hands and eyes and needs to be rebuilt. Talent! Good for Jack! “we are not going to integrate them, they are going to integrate us.”
A good executive wants the problem fixed regardless of who gets the credit. Great story!
.“It is amazing what you can accomplish if you do not care who gets the credit.” – Harry S. TrumanThe David McCullough book on Truman is a work of art.Truman is a poorly recognized and under-appreciated President (1945-53) wedged between FDR and Ike, but he oversaw a lot of history –US use of nuclear weapons, German surrender, defeat of Japan, dismantling of US power after WWII (enormous blunder), Korean War, dismissal of MacArthur, Berlin Blockaded/Airlift, Red China winning their revolution, sending Geo C Marshal to China as Emissary, the Iron Curtain, Marshal Plan, Truman Doctrine, integration of American sports, veto of Taft-Hartley, creation of the NSC, the CIA, the Dept of Def, recognition of Israel (the USA was the first to recognize Israel), military desegregation/integration, Truman Fair Deal, formation of NATO, minimum wage hike from $0.40 >>> $0.75 (this was huge), McCarthy Era, seizure of steel industry to block strike.Truman is, easily, the most underestimated and unstudied President in my lifetime.Of course, I was only two when he left office.JLMwww.themusingsofthebigredca…
“dismantling of US power after WWII” – How so? The US has military bases everywhere.”integration of American sports” – What’s this?https://en.wikipedia.org/wi…https://en.wikipedia.org/wi…
.At the end of WWII, the US Army, Navy, Air Force was tired of the war. We had built a 90 division Army and a total miltiary force of 13MM men with the most horrific combat power imaginable. While the Russians get the credit for wrecking the German army, the US had the best combined arms force — Army, Navy, Air Force and the strongest industrial base. We were the only super power at the end of WWII. We let that advantage slip through our fingers.In the case of Germany, when the Germans surrendered, we abandoned entire Armor divisions in place. Some quarter of a century later, when I was in Germany in the Army in the 1970s, we could still find the old tanks and artillery, where they had been abandoned.We let our cutting edge get dull and when the Korean War started, we had only a couple of fat, lazy, overwhored, overdrunk divisions in Japan as occupiers that were not combat ready. MacArthur, who had seen plenty of Marines in the Pacific War, asked for and got the USMC 1st Div which was the most efficient killing machine ever developed by the USMC.They were not an active division at the time, but the Marines had kept their men on their roles as reservists.The 1st Mar Div was reconstituted from the US without a stick of division level training and sent one regiment to the Pusan Perimeter and loaded out the rest of the division and went from the US to Inchon for the landing with a single stop.Gen MacArthur knew how good the Marines were. The Marines had forced all of their WWII vets to stay in the Reserves, so when the 1st Mar Div got the call it was peopled by WWII vets, seasoned, salty, killers. Most of the officers stepped back into the same jobs they had had on VJ Day.The Div Cdr was Oliver Prince Smith, the best div cdr ever produced by the US.https://themusingsofthebigr…MG Smith fought the brilliant Inchon landing, the seizure of Seoul, and the disastrous Chosin Reservoir fights. His brilliance allowed the 1st Mar Div to escape an attack at the Chosin when outnumbered by 300,000 Chinese. He got out with all his wounded, all of his equipment, and buried all of his dead.The sports analogy is the famous Jackie Robinson, the American pioneer black baseball player.JLMwww.themusingsofthebigredca…
“Nobody cares how much you know, until they know how much you care.” ~Theodore Roosevelt : )
We could use a dose of that mid west calm and plain speaking right about now. I’m a big McCullough fan and will be heading to the library for the Truman book, thanks.
Emphasis on “they instrumented the entire system” – twice I had the opportunity to successfully work with engineering leaders on this type of problem and both of them started just as Greg did with measuring and instrumenting – that prioritize the “grinding” efforts and shuts down the competing “silver bullet” points of view on quick fixes
Looks like the right person (team), right time, right decision.
….and, it’s hard to hide talent.
Completely agree. This motivational equation hangs in our bathroom! https://uploads.disquscdn.c…
Wait let me pull up my calculator 😉
A little something I wrote about this eight years ago, re: Jay Virdy and team: https://www.businessinsider…
Agree, grinding is focused work, often the best path and many time efficiencies are built through this method, which I describe it as the complete dedication approach. It’s also worth noting, this approach works well when a real chess player (strategic thinker) managing the team; driving not just further or faster but focused on effectiveness.
This is a very re-assuring post
ForInstead of searching for a magic solution, they instrumented the entire system and just started rebuilding every part that was about to break. Yup!For a car that doesn’t run well, can get a can of Miracle Magic Super Fix-It Engine Treatment, just $99.95, and pour it in! A person can go to a health food store and hear in serious tones from the clerk that in the fall one should take vitamins B, E and D along with his unique, patented Miracle Magic Super Fix-It Wild Mushroom Extract, just $35.95 a day.Or for the car, MONITOR, e.g., pop the hood, take some measurements, lift the car on a rack, inspect underneath, etc. and DIAGNOSE the darned problems. [I’m a computer guy but I used to be a car guy and was made a Full Member of the SAE.] For a person, go to a good physician who will run off as many $100+ tests as you can afford and DIAGNOSE the darned problems.For a computer system, yup, if it is already in production, at least MONITOR the darned thing.For my software, the code is well set up to scribble a LOT to a log file. I’ve been using a log facility built in to Windows 7, but soon I will take the code for my Web site session state store, rip out most of it, insert a print statement, and get a darned good log server. For the places in my code that can or do write to the log file, they won’t have to be changed but just will call a new version, same name and argument list, of a certain function. And for some such things, I have programmed ways to turn off/on such logging. And it’s easy to do a LOT more just with my code. And supposedly Microsoft has monitoring facilities that can watch nearly everything in real time with lots of associated tools. And when I was in an AI group at IBM’s Watson lab, we were trying to do monitoring. I regarded the AI stuff as total junk-o and stirred up some solid, new, powerful mathematical statistics for some high quality monitoring in quite general terms and published the work in Information Sciences. I intend to apply that work for high level monitoring of my server farm.So, right with monitoring can do detection, gather data, do diagnosis, and then do correction. Standard three steps. And, yup, the monitoring should provide LOTS of relevant data easily with some flexibility to have the data gathering turned off/on during production, etc.Of course, in development, e.g., as for a car or most development, should ENGINEER the darned system, one part at a time, then one subsystem at a time, etc. E.g., Chevy, Ford, and Dodge have been putting superchargers on some of their old V-8 engines and getting engine torque up to ballpark 700 foot pounds. Okay — that took some engineering in the engine. E.g., at least for a while, Chevy went for titanium connecting rods and had oil spraying on the insides of the cylinders. Well, the 700 foot pounds is also important for the frame, the engine mounts, the transmission, the drive shaft, the differential, the axles, and the rear suspension, at least — each of those needs to be ENGINEERED with applied math, computing, and lots of actual testing and then manufacturing quality control, etc.Gotta be much the same for building software. Should divide the system into relatively independent parts, so divide those, …, then design, ENGINEER, and build from the bottom up the tree of divisions and test and have monitoring at each level. And for something for a large organization, there’s MUCH more to do in testing, versions, etc.That Twitter patched it up on the fly is good!My advice: That was very good luck. Don’t count on being that lucky again! Instead, ENGINEER the darned thing!
.I have done a couple of successful turnarounds wherein I bought chaos with an eye toward creating value by creating order.I renovated thousands and thousands of apartments that had been on the brink of foreclosure/foreclosed, institutionally owned, neglected, and abandoned.I did the same thing thousands of times until the organization could do them blindfolded at a high level of quality, expertise, and cost effectively. By the 1,000th apartment, we were assassins. We just kept doing it for thousands more.Whenever you make order from chaos, you create value. When you create enough value, you get to go to the pay window.Operations has always been about organization, hard work, grinding it out.What makes elite military units special? It is not the doctrine, the people, the weapons — it is the battle drill. Battle drill is the learned response to danger in terrifying situations which is acquired by practice, practice, practice such that it becomes second nature to perform thusly. It is a learned response. It takes time. It is a grind.I recall taking my first patrol out. I was an Airborne Ranger, graduate of a military school, son of a career soldier. When I gave the 5-paragraph field order, everybody listened attentively. (It was a damn good order, and I think some of the men wanted to applaud, but I was a green Lieutenant, so maybe not.)The patrol sergeant looked at the men and, one-by-one, made them jump up and down.I had no idea what he was doing. They never taught me to do that at Ranger School.He listened. When somebody’s rifle sling jingled, he duct taped it stiff. When somebody’s canteen gurgled, he filled it. When somebody’s ammo made a noise, he made them stuff more clips into their ammo pouches until they were silent. He made every soldier who had a cough take a handful of lozenges and put them in their breast pocket. He was a grinder.I asked him, “WTF, sergeant?””The enemy can hear everything, Lieutenant. If they hear us, they will come kill us.”Grinding means paying attention to everything.When I figured it out, I would say, “Check, double check, re-check . . . everything.” I did the same thing in business.Grinding is operating. Always has been.Want a good book — The Checklist Manifesto, Atul Guwunde. A good checklist will help you grind. Hard.JLMwww.themusingsofthebigredca…
So will a good hooker
When I was commenting earlier, I thought of you. When I once asked you how I could perpetuate my business through others, you introduced me to the concept of checklists and recommended The Checklist Manifesto. It is the moments of brilliance that the checklists can’t emulate, but the things on the checklist represent 80% of the success so that is a good place to start.
This mirrors something I have been saying to myself lately: “the answer is always more work”.I think there is this perversion to think there is an efficiency optimization or growth hack to most problems. most times it is just brute force of the thing that is already working.
Completely agree. Time + execution. The only downside is when you are short on time or lack executional skills.
There are several points you have made over the years that have stuck with me, and that line from Jack Dorsey that “we are not going to integrate them, they are going to integrate us.” is at the top of that list. I think of it often.But this posting makes me feel old. You wrote original article in 2010.
@Fred. Curious if you feel the same way about building a market or getting to product market fit vs fixing tech in a breakout situation like Twitter.
Grinding. Slogging.As someone who works on a project basis, I continuously relive the cycle. There is always a time of grinding it out , slogging through… before getting to success. The knowledge that we will get there helps in getting through the grueling times.Having the process in place that always seem to work… mixed in with those moments of brilliance when a new idea leads down another path that turns things around..Maybe one of the things that makes someone a good business leader is a pattern of those moments of brilliance that overlay the well-developed process that grinds it out.