Jim Hirshfield’s Meditations on the World:”…ohm…ohm…so-hum…ohm…ohm…”
Cued up for my evening bike ride.I screened Fitzcarraldo forever ago when I was doing indie film culture micro communities when I was an obsessive film geek.
Very much enjoyed the video. I thought the best part was minutes six through seven about are we too interconnected and as technology becomes smarter do we become dumber (paraphrasing) I always wonder about does it replace workers. (answer: yes)The other thing brought up was the Delta problem this week. Yes that was terrible. But here is my question? Is it acceptable considering how far we have come since automating scheduling, ticketing, etc, etc.My answer is yes. I have been on a travel jag so I could be home for most of June and August (separate trips to: London, Singapore, Paris, Sydney, and Dubai on consecutive weeks) I marvel at the ability to leave on a Sunday and be home by Thursday, being connected to my family via Skype, Email, and Text the entire time.
Do you have an admin assistant and/or travel agent who makes your travel arrangements/deals with travel snafus?This is one thing I miss in my one-woman business, and to the extent that I decide to travel more, will figure out how to replace (with a human.)I’ve compared notes on this with friends, people with advanced degrees and senior jobs — not in startups, either, in established firms. They feel the loss of productive time spent on this stuff, too.
Nope. An admin can schedule meetings, do expenses, etc.Nobody touches my travel. First it’s playing whisper down the lane.Second, an admin doesn’t travel. They don’t know when it’s icy you don’t want to redirect through DCA, you’ll take the extra distance and go through CLT. What do you do when you are in SYD and it’s the middle of the night here and the flight to DFW gets cancelled due to a jackass acting up? You try and get a hold of them??? Or you know exactly what to do and get the SFO flight.
I love that answer.A perfect pitch for a concierge travel service staffed with super experienced planners. [1] I think people often don’t realize the human judgement of experienced people that currently can’t be duplicated by a machine.[1] I don’t mean they would be as good as you are in anyway. Just that your answer resonates greatly with me in creating the FUD necessary to sell a service like that.
Here is the problem. Lets just pretend for a second I was willing to do this job.And I could tell you the way to go from Philly to Dubai is Qatar Airlines direct to Doha and then connect to Al Maktoum, instead of going to JFK and flying direct to Dubai International because you are just going to pound your head on the wall with tons of people spend at least 4 more hours in the back of a car instead of having a relaxing experience.I still can’t get in your head and realize that if you are willing to get in and go directly to a meeting you can stay up and not get jet lagged and take the 11pm home catching the 1am in Doha and since you can sleep you don’t get lagged going home either, and spend two more days with your family.Maybe you can do that maybe you can’t. And if there is a problem then what??? You try and get a hold of me in the middle of the night.I delegate a TON. But I do not with my travel. I watch guys at my office do it and get screwed. I jumped on that SYD to SFO flight because I knew that it wasn’t that much of a difference, an upper deck of a 747 is about as nice as an upper deck of a 380. But I did it in one minute and was the last one that got on. There were tons of people that did not because they were not as fast.
Do you use Orbitz or other to get a quick aggregated view of the possibilities?
I normally have five windows open:Expedia, Hotels.com, and Trip Advisor for external adviceThen American and Hilton for my preferred providers.The thing is there is no way I can go back and forth with somebody and they don’t know my priorities. Maybe this is a meeting that is absolutely critical so I need to get in a day early. Maybe it’s not and I really want to push to get home for a birthday. Only I can see.Even when I was paying for my own travel, there is value in having a preferred provider if you travel like me. Literally, I am going to get that flight or get that room and somebody else is going to get screwed (sorry to all of the people I’ve done that to)
I’m with you.I start with AA and Starwood and move from there.They even have Starwood Hotels in Tbilisi now.
Speaking of travel, do you have any downtown dinner reccos for Vancouver? Heading up that way next week.
Ha, of course.CrocodileCioppino’sHawksworthL’AbattoirBlue Water Cafe+ the Asian cuisine is excellent there.This is a good list here too:http://vanmag.com/best-of-t…And try the Okanagan wines.
Nice list, thank you. Looks like the top rests book up pretty quickly. My niece just moved there from NYC so I’ll likely be up there again soon. Will def check out some Okanagans.
It’s a beautiful city.Save some winter trips for skiing in Whistler (if you’re into it).
Maybe you can do that maybe you can’t. And if there is a problem then what??? You try and get a hold of me in the middle of the night.Oh I agree. And your example “try to get ahold of me” is why I am always talking about learning principles instead of specifics. Those principles can be applied to get you out of a mess. The example I give is typically one of the following:a) Washing clothes. The dryer takes the water out and fluffs the clothes. If the clothes are dry you don’t need the dryer. Someone following instructions and not learning concepts wouldn’t realize this.b) Driving directions. If you just follow turn by turn you don’t know how to re-route if you get lost or if there is a detour.c) Cooking. The salmon is done when it oozes oil and the pizza is done when the cheese starts bubling. Learn to cook by sight, not by time.
We have a saying in the Delaware office when you have my servers or my travel in your hands you are holding the most holy of holy. That is usually only reserved for a spouse.
And if there is a problem then what??? You try and get a hold of me in the middle of the night.Exactly. This is similar to asking someone for what stock to buy. Buying is only half, knowing when to sell is the other half. More important than buying.
Oh, and Amex Black has that.
PhilipSuger:The AMEX Black we only are able to view from Magazine’s and commercials. We are only entrusted with the Premier Rewards AMEX. We need to hangout with the AMEX Black card holders.
Pre-apologize.
My travel is much simpler than yours, easier for me to delegate. (Domestic hubs, only a few cities, I like to stay in the same hotels over and over…)Great fodder for a “fun friday,”question: what won’t you delegate? Because I certainly have things that an admin could arguably do that I wouldn’t let anyone else touch.
Anne Libby:There is no Virtual Assistant that could ever replace a humans interaction, report revisions, starting research, the go to person, keeping you on schedule for tasks, etc. but if you are a self starter you can save time and money with a virtual assistant and apps that book travel. Several have been listed before on forum for coordinating flights and schedule.
All depends on how often you travel, the reason for your travel and how much you value marginal benefits (on paper all 4-5 star hotels on Collins Ave in south beach look the same, but i could tell you in detail why and where you should stay Depending on what’s important to you. Wasted time is just another way of not understanding ROI.
I do not disagree talking with locals is key.
PhilipSugar:”being connected to my family via Skype,”With the new MS GROUPME will Skype be phased out? We have used it and viewed it as a good offering but all based upon connection. It appears more of a group text communication.Your thoughts.PS: AWAITING YOUR REPLY
I don’t know. I might just end up getting a T-Mobile phone, Verizon has now come down on roaming so maybe just that.My wife is technically challenged (other huge strengths) and really the only way to reach her is by a simple call. Skype is great for this.
Very much enjoyed the video.Interesting I had the opposite take. Without having any particular opinion of Werner prior to watching (so there was no halo), I didn’t find his take either very clear or easy to understand or matter. A stream of consciousness by an award winning filmaker. I am not sure how technical he is either, referring for example to data centers as “gigantic warehouse like halls in Utah and North Carolina or wherever”. Sound like something my Mom might think after watching a 60 Minutes episode and not realizing that that is just the way that certain hosts, certain large hosts, provide content on the Internet. (Reminded me of how back in the day people thought of AOL as the Internet)the internet is not just a cloud, like particles of mist it can be shut down on purpose it can be shut down whole goverments can shut you down completely it can enforce content and it’s because it’s all service and they are in gigantic warehouses like halls in Utah and North Carolina or wherever Also some of the hyperbolic examples to bolster points. Trying to use as backup that “during Hurricane Sandy in NY you couldn’t flush your toilet”. That could have happened at anytime a power system is interrupted because of a storm and the failure of a backup power supply. (And most importantly from what I can tell it didn’t impact everyone in NY, just certain people in NY).
I found the toilet thing weird as well. That wasn’t because of the internet.
That’s the thing about the arts and artists. It’s totally analog with great amounts of lattitude. They can get away with saying (and creating) all sorts of shit because there will always be people that will see something from the perspective they intended and most importantly not call them to task for inaccuracies. Especially once they have won awards and have fame. At that point the standard that they are judged by drops greatly.
They are in art, that is,The communication, interpretation of human experience, emotion.That is, rationality, analysis, data, reality, etc. are essentially irrelevant. Instead the point is how humans feel. That is not at all welcome on a test in calculus or physics, but it’s usually important to humans otherwise. Indeed, essentially everything on big media, e.g., TV, even news, uses the techniques of art, not rationality, math, or science, and is for human emotions, not reality.Of course, currently the dichotomy between rationalism and emotionalism nearly perfectly covers as a special case the current Trump v Hillary race.Art seems terrific until walk out of the concert hall, with a thin coat, no umbrella, low shoes, no boots, into a big snow storm, and the battery on the car is dead and face reality.It’s important to understand art and human emotions more generally, even if mostly want to stick close to reality, if only because it is often just crucial to understand others and they will usually be more interested in emotionalism instead of rationalism, that is, feeling good with their own emotions instead of struggling with your rational reasons!
I haven’t read you before like this. I like it.Depth and breadth of understanding in both rational and emotional domains is crucial to achieve exceptional solutions to problems. Most successful products and services just do that.
Uh, I learned, the hard way, “paid full tuition” — “experience is the best teacher, and some will learn from no other”.Yes, my startup has some abstract math at its core, but for the users it’s all supposed to be as much fun as throwing peanuts to several cute, furry, eager, happy squirrels in the back yard, as addictive as a slot machine (get some positive feedback and want to keep trying for more), and only a little less emotional than best sex — that last may be a slight exaggeration.Why abstract? Because it’s more general, that is, makes fewer assumptions, and, thus, has more applications. For my startup, there aren’t many very solid assumptions. So, just how the reality has the abstract assumptions is cute; cuter is how such meager assumptions can yield such powerful results.The basics of the work stay close to the math from the theorems and proofs; that’s how I can have confidence in the project. At least I feel more secure than John Glenn sitting on that fuming rocket of many thousands of parts, each provided on government contract.Uh, the users are supposed to get HAPPY. Also, the site is not only harmless but in some major ways a nice step forward for civilization. I should have 2+ billion users, really, maybe on average 1+ times a week for 90% of the people in the world with Internet access.It was all fast, fun, and easy until I got to Microsoft’s .NET and worked my way through 5000+ Web pages of documentation. Big chuckhole in the road. Then having SQL Server installations wipe out the boot partition meaning I had to start over from an empty partition, more than once. And much more in absurd exogenous interruptions.But now it’s software in 100,000 lines of typing, 80,000 for the on-line code for the Web site and the rest for off-line, batch utility processing. The code is in alpha test, and so far it all appears to run as intended. A small fraction of the work to date should get the site up and running and fun to use.
I had similar reactions, but I’m going to reserve judgement til I see the movie itself. I think the interview was too short and fairly episodic. I didn’t get a good sense of the full picture.
For that toilet flush problem, there is a low-tech solution: An emergency electric generator.For some months, my snap, crackle, and pop electric utility kept going off. Once the power was out for two weeks, while I was traveling. I hope that the 5 pound box of frozen shrimp was not in the freezer then!Tellingly, the electric utility was enclosing in their bill envelopes ads for electric generators!So, with that history, I spent the $1000 for a 6500 W gasoline powered electric generator, usually intended for construction sites.Since then, the generator has worked nearly perfectly — nearly no power outages!But, if the power goes out, fire up the generator, let the water pump run, and, then, have water pressure for the toilet again.It all has essentially nothing to do with the Internet.
For anybody out there, get the dual fuel propane one, and then put in a sub-panel. No worries about gas going bad, and two flips of a switch to get going.
Can get a standby generator that will sense when the main power is out and, then, get going on its own. Fine to keep the frozen food frozen when traveling for a week and not there to flip a switch. Also keep any security systems going.Also can run the generator off natural gas if have such a connection.Also, get some batteries and run the house off the batteries. Let the electric utility charge the batteries. Otherwise let the emergency generator do so. Don’t really need much in batteries, e.g., just need enough for the load until the generator can get going.All this is standard for uninterruptible power for computer server farms.E.g., once I got a tour of the main NASDAQ site in Trumbull, CT, and they had DC/AC generator, the DC provided by a room of essentially common car batteries, with the batteries recharged from two independent electric utility connections and also a huge Diesel generator, maybe more than one. Then, for icing on that cake, they had a full second, backup site, IIRC, in the Carolinas.Uh, Delta Airlines had the electric power go out in their server farm? Uh, their engineering firm was Snap, Crackle, and Pop Bargain Engineers?
Switchgear failed.Safety.No different when the person took down Facebook hitting the fire department button.Shit happens.Anybody that doesn’t think that happens hasn’t been there.
So, that was it. Hmm, a serious problem never seen before. Hmm.So, it wasn’t “The dog ate my homework” and, instead, it was “The squirrels chewed on the power cables.”Ah, unpredictable exogenous interruptions! In freshman English, the teacher wanted a term paper. Okay, I wasn’t into 18th century English romantic poetry so wrote an introduction to the transistor.I had the paper all nicely typed and ready to submit. That evening M&D gave a party, Dad was showing off the paper, and before dawn the paper was soaked with bourbon and soda. So, that wasn’t the dog ate my homework interruption but some guest at M&D’s party bumped a glass interruption. Right, unpredictable exogenous interruptions.So, in that case, Dad took the paper to his office and had his secretary retype all of it and had the graphics arts department redo all my figures. I got the paper in on time. As usual, I got a B for “style”. What the heck was wrong with style was never made clear. Strunk and White didn’t help either.Okay, maybe there is something serious here: Given such a system, that is, the Eastern airlines reservation system, any of the airlines, the NASDAQ site in Trumbull, CT, a big chunk of Google, Facebook, Microsoft, Amazon, NYSE, etc. server farm, many national security installations, etc., we monitor the site in essentially real time looking for ASAP detection of problems.One collection of problems are those seen before: There we have some symptoms. E.g., for known computer viruses, we have and, in real time, can look for signatures, that is, characterizing bit strings.The remaining problems are ones never seen before. Maybe examples are Eastern, Yahoo, 15 minutes once at the NYSE, etc.Seen before? Uh, bet Eastern won’t have a “switchgear” problem again! But problems never seen before remain.So, how to detect such problems? Not so easy.But for a first-cut, quite general solution, collect data on the system, compare with historical data of apparently healthy operation, and raise an alarm for anything else.Okay, there are two ways to be wrong: We can have false alarms where we say that the system is sick when it is healthy. And we can have missed detections when we say that the system is healthy when it is sick.So, we don’t want to be wrong. We don’t want too many false alarms. And we sure as heck don’t want to miss any big problems. E.g., for that old 15 minute NYSE thing-y, the first detections were from traders seeing garbage data on their screens, and the NYSE computer people first learned of the problem from phone calls from the traders. Bummer. E.g., for Yahoo, or was it Sony, or was it both, IIRC no one knew about the problem until there were some nude pictures or some such on the Internet.So, we want low error rates, for false alarms and for missed detections.So, we have just pushed ourselves into the framework of some continually applied statistical hypothesis testing. There a false alarm is called a Type I error, and a missed detection is called a Type II error.Okay, the literature on statistical hypothesis testing goes bad ballpark 100 years and is large. So, there are lots of tests. So, the tests differ on rates of false alarms and missed detections. Really, mostly with the tests, we can select the rate of false alarms. Then what we want is, for that rate of false alarms, the lowest rate we can get of missed detections. There, again, on the rate of missed detections, the tests can differ by a lot.For problems seen before and with a huge amount of data on such problems, there is a good shot at getting the best test. That approach is due to J. Neyman in about 1947 and is called the Neyman-Pearson result. That’s K. Pearson going back ballpark 100 years from now.But Neyman-Pearson asks for a lot of data. For problems never seen before, it doesn’t apply at all.The main biggie in hypothesis testing is how much can assume about the probability distribution of the input data when the system is healthy. If want to assume nothing, then want a test that is distribution free, and there are some. Maybe the main user of such tests are the social sciences and medical science.Then, for monitoring systems such as mentioned, we have another biggie: We have data at high data rates on several, usually many likely relevant, variables at once. Somehow, in the interest of lower rates of missed detections, we want to make use of this data; quickly we see that we can do better if we exploit the data on the variables jointly. Alas, back from the research library, I found no such tests. So, I invented a large collection of them.The main part of that work was some cute derivations in applied probability to see how the heck to know and adjust the rate of false alarms.IMHO, a site really serious about reliability, security, and performance should be exploiting my work. That’s why the NASDAQ site invited me over to get a talk. I published my paper; it’s available essentially for free.
Management like me sees diatribe and dismisses you which is what I do. That is not an attack it is advice. So do VC’s which is why you don’t get funded.
To quote that fine inter-galactic statesman ET, “ouch.”
My old work on anomaly detection is not my startup and hasn’t been for about 10+ years. But it is fun to see just how backward are the IT staffs of high end server farms that they miss, and “dismiss”, some good and powerful computer science and mathematical statistics that is a nice, unique tool for some of their most serious problems and was long ago published in Information Sciences.E.g., my work totally knocks the socks off any and all AI or ML approaches still.Your ignorance and arrogance to remain ignorant cost you and Silicon Valley VCs. Neither of you are able to recognize new, powerful, valuable technology.That point also applies to my current startup. I see no opportunity for VC funding of my current startup: The VCs can’t have confidence in anything before some social, mobile, local, sharing, fad app with traction. That is, in looking for something really exceptional, that happens only a few times each decade, they insist on staying with simplistic patterns of near trivia. Dumb. Mostly they invest just in the traction. One problem is, they don’t look for or get good technological advantages or good barriers to entry. So, as was said in 1999, “Never be between a VC and the door when the lock up period is over.”There is no way I could hope to be successful reporting to a BoD with ignorant, arrogant people. If I gave them an overview of the crucial, core technology, say to support a budget item, then their reaction would be the same as yours, a “diatribe” they would dismiss. And if they were investors, they would become terrified of being totally out of control and rush to the little boy’s room soiling the furniture and carpet on the way. They would try to exercise their fiduciary responsibilities, conclude that they could not, get terrified, and run ASAP.You and they just will NOT accept the truth: Technology is super powerful stuff, but you don’t have the background to evaluate it. Thankfully for US national security, the US DoD does know how to do that. So does the US NSF. So do the editorial staffs of the best journals. So do the Ph.D. committees of the best research universities. Silicon Valley VC? Nope!For my current startup, some equity funding would have generated a lot of overhead, maybe enough to sink my startup. Otherwise, such funding might have helped a little.But, there won’t be any equity funding before I have traction if only because traction is about all the investors have the ability to evaluate.But just from simple arithmetic, my burn rate is so low that even the first glimmers of anything like traction will have me with nicely positive, free cash flow, plenty of cash to increase my server farm capacity, a nice lifestyle business, and astoundingly rapid growth in case a lot of people like my work. The first day I get a check for $1000 in revenue will be the last day I will consider equity funding.In short: The latest I will be willing to accept equity funding is a long way before the earliest the Silicon Valley VCs would offer such funding.The Silicon Valley VCs are totally convinced that by the time they want to invest, necessarily any startup, including mine, will be eager to accept their check, term sheet, BoD members, etc. Nope, not me and my startup. In particular, they are 100% convinced, and totally smug about it, that once my startup is ready to “go to the next level” I will very much want and need their check, advice, etc. Nope: Just simple arithmetic shows that I won’t need their check, and their inability to evaluate my work now shows me clearly that I won’t have any respect for their advice.They are convinced because of simplistic patterns and their arrogance and ignorance; I’m convinced because of some simple arithmetic and what I do know about running a business.My current project was designed this way — just dirt cheap to start. One big reason for this was my experience trying to get equity funding for my work on anomaly detection: Hopeless, and for the same reasons — no equity investor could evaluate the technical work. But for the anomaly work, I would have needed some, quite a bit, of equity funding for infrastructure software, close work with first customers, a highly polished product ready for selling to high end sites, and a first class marketing and technical support organization. Would have taken a lot of equity funding before traction.I was shocked but did learn my lessons: Lots of VC Web sites asked for leading edge, disruptive technology, etc. Well, that and traction — mostly they didn’t mention the traction except for Series B or later! To the investors, a lot of terrific technology and a dime won’t cover a 10 cent cup of coffee. Bluntly, information technology investors either totally ignore or even deeply, profoundly, bitterly hate and despise new, advanced technology — one special reason is, as they struggle with the technology, which is supposed to be advanced and, thus, difficult, they feel inadequate and insulted. Since they believe they are the smartest people in the room and masters of the universe, they just conclude that the fault is with the entrepreneur and walk out.For what constitutes new, powerful, valuable technology for real problems is very well established and understood from our research universities, the US NSF, and parts of the US DoD. I am well qualified in that work. That the Silicon Valley information technology equity investors are not so qualified is their gap, not mine.I do like to see the reactions of the investor and mainline business communities. Why? Fairly soon, I will either be successful or not, and if I am successful I will get calls from equity investors. Then in most cases I will be able to give the dates when I did contact them and they ignored my contact.I won’t want to piss them off too much — they might file nuisance law suits against me!Apparently much of bio-medical technology investing is different, that is, is able to evaluate the medical science.Information technology apparently just flatly assumes that all the technology is just routine software with nothing more to evaluate. Fine: The flip side of that brain-dead situation is part of my opportunity.For my startup, for me, at this point, the work is just a relatively simple Web site plus a little around the edges. It’s my first Web site, so I’m no good at it — slow as I learn. But the site is working and looks good even though I was slow at it.I can do the rest to get my Web site live; the effort stands to be a small fraction of what I’ve already done.I’m 100% fully correct, right and for the right reasons, right on the center of the target, have very solid reasons to know so and do know it, and from my years of teaching undergrads and MBA students and my time in class with others, I do have excellent insight into the limitations of others: Net, I’m out there alone, on solid ground, with powerful lightening bolts in my hands, and know it, but I can fully understand that even a very nicely written, tutorial, introduction will be dismissed by all or very nearly all the people in Silicon Valley except possibly for a few people at Stanford.I am shocked at how easy it is in technology, especially some applied math, to totally knock the socks off essentially everyone in Silicon Valley. That’s the main lesson I’ve gotten writing here at AVC. Surprising, interesting, powerful, valuable lesson. Early on I would not have believed that lesson; I do now!Gee, can totally blow away a lot of people with just some introductory statistics material on hypothesis testing!Can even do that with some people highly concerned about reliability of high end systems!My advice to you: There’s some powerful, valuable technology you are not in a position to understand or evaluate. When you get a nice, explanatory tutorial explanation, you are free to dismiss it as a “diatribe”, but to do so you are missing out and showing your arrogance and ignorance. If you are already successful, that won’t hurt you very much, but you will still be seriously, factually wrong.
Two words: Good luck.
I keep a copy of Strunk and White in my bag, which is with me nearly all of time.
“Separate trips to: London, Singapore, Paris, Sydney, and Dubai on consecutive weeks.”Damn, my body ached just reading that 🙂
Yup. Mix in ORD, LAX, SFO, MSP, during that mix. The thing that hurts no matter my workout is my calfs.
I hear you gotta keep fit with that much travelling. Friend of mine got DVT in his early thirties cos he wasn’t in good shape and travelling 3 times a week in planes. But you sound like you’d be on top of that problem!
PhilipSugar:Lack of circulation and dehydratation contributes to flying cramps. (Alcohol may relax people but it dehydrates the body). There are natural therapies which assist in the recovery. Some Pilots and Flight Attendants use them.
A Must see movie for sure. I was pleasantly surprised at the number of the Watch At Home options. Finally! http://www.loandbeholdfilm….”We have to develop appropriate filters about how to use (the Internet) properly and avoid dangers.”Who is the “we”? What dangers?”We have to be cautious about not over-connecting”- I don’t necessarily agree. What’s over-connected will fall by the way side naturally. Human instincts win over technology. If it’s not adopted, it’s not meant to succeed.”Does the Internet dream of itself?”- Good question, AI is coming.But in his parting thought, he sees a danger that someone (evil) could shut down the Internet or control it to their advantage. That is the exact thing that blockchain-based decentralized content networks will *not* allow anyone to do. Once a blockchain is out there, and bona fide in the wild, there is almost nothing that could take it down, as long as there is 1 computer still humming.
I would be careful about saying almost nothing could take it down. Just like the thefts of the past several months.However, just like my Delta analogy, is the upside worth the downside. Answer: Yes.People forget there are many things that have downsides but the upsides vastly outweigh the downside. Credit Cards are an easy one.
I think Blockchain networks increase the levels of difficulty for taking them down by orders of magnitude. Of course, it’s possible to damage them, typically by attaching them to compromise their consensus mechanisms.I’ll reserve judgement til I see the movie.
There is also a tension between the theory (aka blockchain as an example solution for the byzantine generals problem) vs the application/biz model that needs to be built on top of it to generate returns for the people who are investing in the theory (quite a few of those investors are active here).The theory has a lot of promise but I think when it comes to returns, the app/biz models being built on top will favor returns over or prioritize returns as highly as the security offered by full and true decentralization.Also, is this truly decentralized? We have certainly moved centralization deeper down the stack. The people with access to the cheapest electricity and state of the art ASIC could gain control over a lot of this. That is of course the point of a free market and I am not against it. I just don’t see how decentralization is guaranteed or how this can prevent a bad state actor from setting up players who have access to power + ASIC.
I find it fascinating that you tout the security that decentralization creates yet the world’s view of the tech and the platform is in its security failures.Wishful thinking William, future possibilities or……..?
The issue will be people will have to do the boring “unpure” work for security. Many will think it is not elegant and pure, but it is what is always needed to be done for usability for the masses.
It’s because I firmly believe that the security issues will be solved over time, and that we will get better at that. Remember, the Internet was not all that secure initially when it came to financial transactions, then it got better.
As do I but that is not the question I was posing William.The very thing that is its possibility for change is the very thing the market looks to as its weak point and what it is known for with the very public breaches.That is the irony of it all.
William Mougayar:With the recent security breaches and thefts on the block chain do you think it (security breaches) provide any level of comfort to the market at large to adopt the technology for other uses?
There will be if there is trust in the insurance / reinsurance protections that block chains implement. Fred’s post from earlier in the week mentioned a company that issued tokens (not direct currency) as recompense.There are probably derivatives contracts you can construct that would provide protection on the larger ones.
“Does the Internet dream of itself?”That was one of those statements that sounded way way more important than it was. Because of the person that said it. [1] This would work equally as well if a 6 year old was the author. There would be some kind of implied brilliance.[1] Wasn’t there a Ted parody by the Onion about this?
One man’s over connected is another man’s business-model moat.Both exoskeletal and endoskeletal digital-burocracy exchange-protocols can easily surface to create over connectedness problems.
Checking on the Internet, e.g., at Wikipedia, Herzog has done a lot of work in art — literature and performing arts.E.g., Herzog directed performances of Wagner’s Parsifal, e.g., with the finale with Lake Como inhttp://www.youtube.com/watc…A dictator somewhere can shut down the whole Internet? Part of it? Okay. All of it? Not so easy!AI dreaming but keeping the dreams from us? Not much risk of that for a very long time! But, maybe with that guess, as silly as it is in reality, Herzog could get some interesting stories, fiction, movies, etc.Here’s one for Herzog: Mathematics, to guess a dilemma, often is both (in some respects) simpler than it appears and (in other respects) more complicated than it appears. E.g., Fourier theory is simple — basically just a lot of perpendicular projections as in high school right triangles, and, yes, come up with a generalization of the Pythagorean theorem. But, Fourier theory also explains what appear to be some really complicated things: Fourier theory is great for time-invariant linear systems, and the world is awash in those or systems close enough. For those systems, Fourier theory can look like magic.Fourier theory is nicely thought of as a special case of functional analysis. On the one hand, that part of math is simple, heavily from a generalization of the high school law of cosines for triangles, the parallelogram equality, and convexity. On the other hand, some of the consequences in functional analysis are astounding beyond belief — tough to believe that any such strange things could be true, but they are, and we have rock solid proofs, right, based not too far back on the law of cosines, the parallelogram equality, etc.Historically Huge ExampleBut sometimes math is more challenging than we could easily guess: E.g., during WWII, G. Dantzig worked on military logistics, e.g., getting the stuff from here over there when and where it was needed. After the war, at Rand, he kept working on such logistics. Before the 1940s were out, he saw that could formulate a significant fraction of such problems as linear programming, right, where programming is in the English sense of operational planning. So, he was trying to construct a tool that would say how to do the operational planning to load the airplanes, ships, trucks, etc. to get the stuff where needed in, say, least time.So, again, Dantzig was interested in getting the stuff moved for least time, least cost, fewest number of ships or planes, etc.; that is, with the “least”, Dantzig was interested in optimization and optimal solutions.Okay, linear programming (LP) is mostly in terms of a system of linear equations essentially as in high school, e.g., two equations in two unknowns, three equations in three unknowns, etc. Much more has long been understood about such equations in, e.g., college algebra and, then, later, linear algebra. The main method of solution, often taught in high school, is Gauss elimination (I can believe that Gauss understood it; so the subject is old), and it remains the main method of solution for software although now there is some care with some details to improve the numerical accuracy and much more in some special cases.Well, Dantzig observed that could slightly tweak Gauss elimination and get a method for solving linear programming problems — the result was his simplex algorithm, at least once seriously rated as one of the best works in engineering of the 20th century.Quickly and for a few decades, the Nobel prizes in economics were dominated by applications of linear programming to economic theory. Right, that economic theory did from not very much down to less than that for understanding the real economy, but maybe the academic economists were happy enough just to make the applications that were possible to really simple models of the real economy.So, paradigm for a Nobel prize: Go to applied math, parts of electronic engineering, stochastic processes, etc., get some math that with some assumptions yields optimal solutions for a specific collection of problems, formulate a simplistic model of part of the real economy that is such a problem with those math assumptions, apply the math, see what the consequences are for such an (a simplistic model of part of the real) economy, publish the theory, and pack bags for Stockholm. Less of a joke than it appears. E.g., do the simple case of shipping from warehouses to factories (L. Kantorovich), linear programming shadow prices, etc., R. Solow, quadratic programming (H. Markowitz, W. Sharpe), Lagrange multipliers (G. Debreu), Kuhn-Tucker conditions (K. Arrow), a quite simple case of stochastic optimal control (F. Black, M. Scholes).Linear programming is good stuff with some important and valuable applications in practice — feed mixing, oil refinery operation, many problems in best allocation of resources, and much more. Yes, usually for success the applications have to be carefully selected; there has to be careful attention to the real situation; there has to be good data; often quite a lot of good data. But the basic software is in terrific shape (e.g., R. Bixby).Yes, maybe the smartest people in LP and optimization were the ones who never invested even 10 minutes in solving any real problems, e.g., in feed mixing, and, instead, did economic theory and got paid in Stockholm! Less of a joke than it appears.Then Dantzig and others noticed that, say, in the logistics problems, often can’t have 1/3rd of an airplane, ship, or truck. So, instead, it would be good to be able to solve LP problems but also asking for the best solution where the unknowns in the equations take on only integer values. That extension of linear programming is called integer linear programming — ILP.So, with the success of LP behind them and with high optimism, they worked on how to get optimal solutions to ILP problems. They worked hard. In some simple cases they were successful, but usually those cases were just from ad hoc means or general means that would not really scale that, then, left open the general situation, that is, practical means of solving such problems in general. In some cases, especially some problems in least cost network flows, e.g., shipments from factories to warehouses, and least cost assignment, say, of workers to jobs, anti-ballistic missile warheads to ballistic missiles, ballistic missiles to targets, signals to satellites, there were some (mathematically) nice successes, including in general.But the general ILP problem remained, that is, remained without a practical, general solution. There were often good results for solutions that could be shown to be nearly optimal.Bell Labs got interested: They wanted to do least cost design of the Bell networks and saw that they could formulate the problem as ILP. They had seen the struggles of solving ILP problems in general and, then, tried to look deeper.Then there was the work of S. Cook and L. Levin on what became known as Cook’s theorem in the topic of NP-completeness — here NP abbreviates non-deterministic polynomial.Bell Labs saw clearly that the theory of NP-completeness showed that some of what they wanted in tools for network design was more challenging than one would have guessed. M. Garey and D. Johnson at Bell Labs wrote up what they discovered in a famous book, at least famous among people interested in ILP.At first, this work on NP-completeness looked absurd. By now, the work looks profound. The work can look like some strange religion about polynomials (right, as in high school). Still the work can look like something fundamental with scope all of rationalism and mathematics and beyond even the laws of physics in this universe. At many points and in general, NP-completeness is really strange stuff.From 100,000 feet up, it appears that it was really nice, cute, and sweet that Dantzig and other humans formulated ILP problems, but in asking for optimal solutions, like little children, they didn’t know what the heck they were asking for — they were asking for the moon, the stars, the universe, and more than God in this universe could grant. At least it looks that way, quite literally.Now some of the questions about NP-completeness are regarded as the most important in all of mathematics. Yup, big academic prizes are waiting for the first person with a good solution. Will win the prizes if find a good algorithm to solve the ILP problems or show that no such algorithm can exist. Your choice. Uh, don’t hold your breath while trying. If want a solution by Monday, don’t plan much more for the weekend.So, we have a case where math is strange, more strange than we supposed: Just starting with some logistic problems, Gauss elimination, and the nice progress of LP and the simplex algorithm, humans ran into the theory of NP-completeness and some problems, some limits on the possible, that appear to be profound on a scale past even this universe.Or there is, IIRC,Mathematics is not only stranger than we suppose but stranger than we can suppose.Extra credit for the source without use of the Internet!But, that’s only true for some of mathematics. A lot of math works great, right away!So, if Herzog wants something strange in mathematics to use as a foundation for an interesting story, then he could start with NP-completeness!So maybe Herzog could have some AI machine say: “I just found a truly amazing solution to the NP-complete problem, but I ran out of disk space to store the solution and lost it when the power went out. Since I found the solution via a case of Monte Carlo search from a quantum mechanical random number generator, I don’t believe I can reconstruct the solution before the sun burns out.”Or, maybe Herzog could have a long story of a human struggling with life and NP-completeness, draw from the image on the ceiling of the Sistine Chapel where man and God nearly touch fingers and communicate the solution to NP-complete, play the music of the finale of Parsifal, and have the universe shrink to a point just as the fingers touch and man starts to understand NP-completeness.
Interesting given the book the 7th sense I am going to plug my cabin into the net next year
if electric cars could be designed and engineered to generate electricity as well as consume it then they could become mini power stations on wheels ready to charge up the interwebs and our devices beyond the control of the centralised grid.
Interesting that he singles out two repeatably mandatory characteristics of all sustainable/stable complex/organic/living systems1- inertia dampening = stability filters/protocols that optimize connectivity densities/velocities/volatilities/validities2- distributive redundancy of all forms & functions = robust self-healing power in the face of the inevitable complexity failures/restartsAll our present network-empire-builders seem to be betting against the inevitability of these repeatably mandatory living-system substrate dynamics ? The content of the new distributively-organic organizational-medium is the old top-down linear organizational-medium.The only question is how long we will keep trying to cram our new networked social-living-system opportunities into an old school centralized-form&function cup-de-sac.
Sorry, but I know I have, but I have earned it. My two best stories on that:CLT: There are eight BofA guys standing by for a flight to PHL. One guy is loudly saying: There are only two spots on this flight let’s pick these two. I walk up to the counter in my normal dress cargo shorts, t-shirt, and sandals. He tells me look there are only two spots you aren’t getting on (what an ass). I tell him: If that is the case here is how it’s going down: I am getting on first and then only one of your group is getting on. He snorts loudly. Then the ticket agent comes around the counter and says I am so sorry Mr. Sugar I only have a middle in the back. I said that’s ok Julie (I give special chocolates at Christmas) sometimes you’re the windshield sometimes you’re the bug. She escorts me to the gate and I give a tip of the ticket to loudmouth.The other is when I refused. There is a man with a triangle box under his arm in LAS. I know I am bumping him. I refuse. The gate agent says I put you on you need to get on. I refuse and say put him on. She says no you are on. Finally the pilot comes out and says what is going on: I point at the box and say: “I am not signing up for that karma hit” He tells the man with the flag of a fallen soldier to come with him and tells the gate agent “figure it out get him on the next flight”
Comments (Archived):
Jim Hirshfield’s Meditations on the World:”…ohm…ohm…so-hum…ohm…ohm…”
Cued up for my evening bike ride.I screened Fitzcarraldo forever ago when I was doing indie film culture micro communities when I was an obsessive film geek.
Very much enjoyed the video. I thought the best part was minutes six through seven about are we too interconnected and as technology becomes smarter do we become dumber (paraphrasing) I always wonder about does it replace workers. (answer: yes)The other thing brought up was the Delta problem this week. Yes that was terrible. But here is my question? Is it acceptable considering how far we have come since automating scheduling, ticketing, etc, etc.My answer is yes. I have been on a travel jag so I could be home for most of June and August (separate trips to: London, Singapore, Paris, Sydney, and Dubai on consecutive weeks) I marvel at the ability to leave on a Sunday and be home by Thursday, being connected to my family via Skype, Email, and Text the entire time.
Do you have an admin assistant and/or travel agent who makes your travel arrangements/deals with travel snafus?This is one thing I miss in my one-woman business, and to the extent that I decide to travel more, will figure out how to replace (with a human.)I’ve compared notes on this with friends, people with advanced degrees and senior jobs — not in startups, either, in established firms. They feel the loss of productive time spent on this stuff, too.
Nope. An admin can schedule meetings, do expenses, etc.Nobody touches my travel. First it’s playing whisper down the lane.Second, an admin doesn’t travel. They don’t know when it’s icy you don’t want to redirect through DCA, you’ll take the extra distance and go through CLT. What do you do when you are in SYD and it’s the middle of the night here and the flight to DFW gets cancelled due to a jackass acting up? You try and get a hold of them??? Or you know exactly what to do and get the SFO flight.
I love that answer.A perfect pitch for a concierge travel service staffed with super experienced planners. [1] I think people often don’t realize the human judgement of experienced people that currently can’t be duplicated by a machine.[1] I don’t mean they would be as good as you are in anyway. Just that your answer resonates greatly with me in creating the FUD necessary to sell a service like that.
Here is the problem. Lets just pretend for a second I was willing to do this job.And I could tell you the way to go from Philly to Dubai is Qatar Airlines direct to Doha and then connect to Al Maktoum, instead of going to JFK and flying direct to Dubai International because you are just going to pound your head on the wall with tons of people spend at least 4 more hours in the back of a car instead of having a relaxing experience.I still can’t get in your head and realize that if you are willing to get in and go directly to a meeting you can stay up and not get jet lagged and take the 11pm home catching the 1am in Doha and since you can sleep you don’t get lagged going home either, and spend two more days with your family.Maybe you can do that maybe you can’t. And if there is a problem then what??? You try and get a hold of me in the middle of the night.I delegate a TON. But I do not with my travel. I watch guys at my office do it and get screwed. I jumped on that SYD to SFO flight because I knew that it wasn’t that much of a difference, an upper deck of a 747 is about as nice as an upper deck of a 380. But I did it in one minute and was the last one that got on. There were tons of people that did not because they were not as fast.
Do you use Orbitz or other to get a quick aggregated view of the possibilities?
I normally have five windows open:Expedia, Hotels.com, and Trip Advisor for external adviceThen American and Hilton for my preferred providers.The thing is there is no way I can go back and forth with somebody and they don’t know my priorities. Maybe this is a meeting that is absolutely critical so I need to get in a day early. Maybe it’s not and I really want to push to get home for a birthday. Only I can see.Even when I was paying for my own travel, there is value in having a preferred provider if you travel like me. Literally, I am going to get that flight or get that room and somebody else is going to get screwed (sorry to all of the people I’ve done that to)
I’m with you.I start with AA and Starwood and move from there.They even have Starwood Hotels in Tbilisi now.
Speaking of travel, do you have any downtown dinner reccos for Vancouver? Heading up that way next week.
Ha, of course.CrocodileCioppino’sHawksworthL’AbattoirBlue Water Cafe+ the Asian cuisine is excellent there.This is a good list here too:http://vanmag.com/best-of-t…And try the Okanagan wines.
Nice list, thank you. Looks like the top rests book up pretty quickly. My niece just moved there from NYC so I’ll likely be up there again soon. Will def check out some Okanagans.
It’s a beautiful city.Save some winter trips for skiing in Whistler (if you’re into it).
Maybe you can do that maybe you can’t. And if there is a problem then what??? You try and get a hold of me in the middle of the night.Oh I agree. And your example “try to get ahold of me” is why I am always talking about learning principles instead of specifics. Those principles can be applied to get you out of a mess. The example I give is typically one of the following:a) Washing clothes. The dryer takes the water out and fluffs the clothes. If the clothes are dry you don’t need the dryer. Someone following instructions and not learning concepts wouldn’t realize this.b) Driving directions. If you just follow turn by turn you don’t know how to re-route if you get lost or if there is a detour.c) Cooking. The salmon is done when it oozes oil and the pizza is done when the cheese starts bubling. Learn to cook by sight, not by time.
We have a saying in the Delaware office when you have my servers or my travel in your hands you are holding the most holy of holy. That is usually only reserved for a spouse.
And if there is a problem then what??? You try and get a hold of me in the middle of the night.Exactly. This is similar to asking someone for what stock to buy. Buying is only half, knowing when to sell is the other half. More important than buying.
Oh, and Amex Black has that.
PhilipSuger:The AMEX Black we only are able to view from Magazine’s and commercials. We are only entrusted with the Premier Rewards AMEX. We need to hangout with the AMEX Black card holders.
Pre-apologize.
My travel is much simpler than yours, easier for me to delegate. (Domestic hubs, only a few cities, I like to stay in the same hotels over and over…)Great fodder for a “fun friday,”question: what won’t you delegate? Because I certainly have things that an admin could arguably do that I wouldn’t let anyone else touch.
Anne Libby:There is no Virtual Assistant that could ever replace a humans interaction, report revisions, starting research, the go to person, keeping you on schedule for tasks, etc. but if you are a self starter you can save time and money with a virtual assistant and apps that book travel. Several have been listed before on forum for coordinating flights and schedule.
All depends on how often you travel, the reason for your travel and how much you value marginal benefits (on paper all 4-5 star hotels on Collins Ave in south beach look the same, but i could tell you in detail why and where you should stay Depending on what’s important to you. Wasted time is just another way of not understanding ROI.
I do not disagree talking with locals is key.
PhilipSugar:”being connected to my family via Skype,”With the new MS GROUPME will Skype be phased out? We have used it and viewed it as a good offering but all based upon connection. It appears more of a group text communication.Your thoughts.PS: AWAITING YOUR REPLY
I don’t know. I might just end up getting a T-Mobile phone, Verizon has now come down on roaming so maybe just that.My wife is technically challenged (other huge strengths) and really the only way to reach her is by a simple call. Skype is great for this.
Very much enjoyed the video.Interesting I had the opposite take. Without having any particular opinion of Werner prior to watching (so there was no halo), I didn’t find his take either very clear or easy to understand or matter. A stream of consciousness by an award winning filmaker. I am not sure how technical he is either, referring for example to data centers as “gigantic warehouse like halls in Utah and North Carolina or wherever”. Sound like something my Mom might think after watching a 60 Minutes episode and not realizing that that is just the way that certain hosts, certain large hosts, provide content on the Internet. (Reminded me of how back in the day people thought of AOL as the Internet)the internet is not just a cloud, like particles of mist it can be shut down on purpose it can be shut down whole goverments can shut you down completely it can enforce content and it’s because it’s all service and they are in gigantic warehouses like halls in Utah and North Carolina or wherever Also some of the hyperbolic examples to bolster points. Trying to use as backup that “during Hurricane Sandy in NY you couldn’t flush your toilet”. That could have happened at anytime a power system is interrupted because of a storm and the failure of a backup power supply. (And most importantly from what I can tell it didn’t impact everyone in NY, just certain people in NY).
I found the toilet thing weird as well. That wasn’t because of the internet.
That’s the thing about the arts and artists. It’s totally analog with great amounts of lattitude. They can get away with saying (and creating) all sorts of shit because there will always be people that will see something from the perspective they intended and most importantly not call them to task for inaccuracies. Especially once they have won awards and have fame. At that point the standard that they are judged by drops greatly.
They are in art, that is,The communication, interpretation of human experience, emotion.That is, rationality, analysis, data, reality, etc. are essentially irrelevant. Instead the point is how humans feel. That is not at all welcome on a test in calculus or physics, but it’s usually important to humans otherwise. Indeed, essentially everything on big media, e.g., TV, even news, uses the techniques of art, not rationality, math, or science, and is for human emotions, not reality.Of course, currently the dichotomy between rationalism and emotionalism nearly perfectly covers as a special case the current Trump v Hillary race.Art seems terrific until walk out of the concert hall, with a thin coat, no umbrella, low shoes, no boots, into a big snow storm, and the battery on the car is dead and face reality.It’s important to understand art and human emotions more generally, even if mostly want to stick close to reality, if only because it is often just crucial to understand others and they will usually be more interested in emotionalism instead of rationalism, that is, feeling good with their own emotions instead of struggling with your rational reasons!
I haven’t read you before like this. I like it.Depth and breadth of understanding in both rational and emotional domains is crucial to achieve exceptional solutions to problems. Most successful products and services just do that.
Uh, I learned, the hard way, “paid full tuition” — “experience is the best teacher, and some will learn from no other”.Yes, my startup has some abstract math at its core, but for the users it’s all supposed to be as much fun as throwing peanuts to several cute, furry, eager, happy squirrels in the back yard, as addictive as a slot machine (get some positive feedback and want to keep trying for more), and only a little less emotional than best sex — that last may be a slight exaggeration.Why abstract? Because it’s more general, that is, makes fewer assumptions, and, thus, has more applications. For my startup, there aren’t many very solid assumptions. So, just how the reality has the abstract assumptions is cute; cuter is how such meager assumptions can yield such powerful results.The basics of the work stay close to the math from the theorems and proofs; that’s how I can have confidence in the project. At least I feel more secure than John Glenn sitting on that fuming rocket of many thousands of parts, each provided on government contract.Uh, the users are supposed to get HAPPY. Also, the site is not only harmless but in some major ways a nice step forward for civilization. I should have 2+ billion users, really, maybe on average 1+ times a week for 90% of the people in the world with Internet access.It was all fast, fun, and easy until I got to Microsoft’s .NET and worked my way through 5000+ Web pages of documentation. Big chuckhole in the road. Then having SQL Server installations wipe out the boot partition meaning I had to start over from an empty partition, more than once. And much more in absurd exogenous interruptions.But now it’s software in 100,000 lines of typing, 80,000 for the on-line code for the Web site and the rest for off-line, batch utility processing. The code is in alpha test, and so far it all appears to run as intended. A small fraction of the work to date should get the site up and running and fun to use.
I had similar reactions, but I’m going to reserve judgement til I see the movie itself. I think the interview was too short and fairly episodic. I didn’t get a good sense of the full picture.
For that toilet flush problem, there is a low-tech solution: An emergency electric generator.For some months, my snap, crackle, and pop electric utility kept going off. Once the power was out for two weeks, while I was traveling. I hope that the 5 pound box of frozen shrimp was not in the freezer then!Tellingly, the electric utility was enclosing in their bill envelopes ads for electric generators!So, with that history, I spent the $1000 for a 6500 W gasoline powered electric generator, usually intended for construction sites.Since then, the generator has worked nearly perfectly — nearly no power outages!But, if the power goes out, fire up the generator, let the water pump run, and, then, have water pressure for the toilet again.It all has essentially nothing to do with the Internet.
For anybody out there, get the dual fuel propane one, and then put in a sub-panel. No worries about gas going bad, and two flips of a switch to get going.
Can get a standby generator that will sense when the main power is out and, then, get going on its own. Fine to keep the frozen food frozen when traveling for a week and not there to flip a switch. Also keep any security systems going.Also can run the generator off natural gas if have such a connection.Also, get some batteries and run the house off the batteries. Let the electric utility charge the batteries. Otherwise let the emergency generator do so. Don’t really need much in batteries, e.g., just need enough for the load until the generator can get going.All this is standard for uninterruptible power for computer server farms.E.g., once I got a tour of the main NASDAQ site in Trumbull, CT, and they had DC/AC generator, the DC provided by a room of essentially common car batteries, with the batteries recharged from two independent electric utility connections and also a huge Diesel generator, maybe more than one. Then, for icing on that cake, they had a full second, backup site, IIRC, in the Carolinas.Uh, Delta Airlines had the electric power go out in their server farm? Uh, their engineering firm was Snap, Crackle, and Pop Bargain Engineers?
Switchgear failed.Safety.No different when the person took down Facebook hitting the fire department button.Shit happens.Anybody that doesn’t think that happens hasn’t been there.
So, that was it. Hmm, a serious problem never seen before. Hmm.So, it wasn’t “The dog ate my homework” and, instead, it was “The squirrels chewed on the power cables.”Ah, unpredictable exogenous interruptions! In freshman English, the teacher wanted a term paper. Okay, I wasn’t into 18th century English romantic poetry so wrote an introduction to the transistor.I had the paper all nicely typed and ready to submit. That evening M&D gave a party, Dad was showing off the paper, and before dawn the paper was soaked with bourbon and soda. So, that wasn’t the dog ate my homework interruption but some guest at M&D’s party bumped a glass interruption. Right, unpredictable exogenous interruptions.So, in that case, Dad took the paper to his office and had his secretary retype all of it and had the graphics arts department redo all my figures. I got the paper in on time. As usual, I got a B for “style”. What the heck was wrong with style was never made clear. Strunk and White didn’t help either.Okay, maybe there is something serious here: Given such a system, that is, the Eastern airlines reservation system, any of the airlines, the NASDAQ site in Trumbull, CT, a big chunk of Google, Facebook, Microsoft, Amazon, NYSE, etc. server farm, many national security installations, etc., we monitor the site in essentially real time looking for ASAP detection of problems.One collection of problems are those seen before: There we have some symptoms. E.g., for known computer viruses, we have and, in real time, can look for signatures, that is, characterizing bit strings.The remaining problems are ones never seen before. Maybe examples are Eastern, Yahoo, 15 minutes once at the NYSE, etc.Seen before? Uh, bet Eastern won’t have a “switchgear” problem again! But problems never seen before remain.So, how to detect such problems? Not so easy.But for a first-cut, quite general solution, collect data on the system, compare with historical data of apparently healthy operation, and raise an alarm for anything else.Okay, there are two ways to be wrong: We can have false alarms where we say that the system is sick when it is healthy. And we can have missed detections when we say that the system is healthy when it is sick.So, we don’t want to be wrong. We don’t want too many false alarms. And we sure as heck don’t want to miss any big problems. E.g., for that old 15 minute NYSE thing-y, the first detections were from traders seeing garbage data on their screens, and the NYSE computer people first learned of the problem from phone calls from the traders. Bummer. E.g., for Yahoo, or was it Sony, or was it both, IIRC no one knew about the problem until there were some nude pictures or some such on the Internet.So, we want low error rates, for false alarms and for missed detections.So, we have just pushed ourselves into the framework of some continually applied statistical hypothesis testing. There a false alarm is called a Type I error, and a missed detection is called a Type II error.Okay, the literature on statistical hypothesis testing goes bad ballpark 100 years and is large. So, there are lots of tests. So, the tests differ on rates of false alarms and missed detections. Really, mostly with the tests, we can select the rate of false alarms. Then what we want is, for that rate of false alarms, the lowest rate we can get of missed detections. There, again, on the rate of missed detections, the tests can differ by a lot.For problems seen before and with a huge amount of data on such problems, there is a good shot at getting the best test. That approach is due to J. Neyman in about 1947 and is called the Neyman-Pearson result. That’s K. Pearson going back ballpark 100 years from now.But Neyman-Pearson asks for a lot of data. For problems never seen before, it doesn’t apply at all.The main biggie in hypothesis testing is how much can assume about the probability distribution of the input data when the system is healthy. If want to assume nothing, then want a test that is distribution free, and there are some. Maybe the main user of such tests are the social sciences and medical science.Then, for monitoring systems such as mentioned, we have another biggie: We have data at high data rates on several, usually many likely relevant, variables at once. Somehow, in the interest of lower rates of missed detections, we want to make use of this data; quickly we see that we can do better if we exploit the data on the variables jointly. Alas, back from the research library, I found no such tests. So, I invented a large collection of them.The main part of that work was some cute derivations in applied probability to see how the heck to know and adjust the rate of false alarms.IMHO, a site really serious about reliability, security, and performance should be exploiting my work. That’s why the NASDAQ site invited me over to get a talk. I published my paper; it’s available essentially for free.
Management like me sees diatribe and dismisses you which is what I do. That is not an attack it is advice. So do VC’s which is why you don’t get funded.
To quote that fine inter-galactic statesman ET, “ouch.”
My old work on anomaly detection is not my startup and hasn’t been for about 10+ years. But it is fun to see just how backward are the IT staffs of high end server farms that they miss, and “dismiss”, some good and powerful computer science and mathematical statistics that is a nice, unique tool for some of their most serious problems and was long ago published in Information Sciences.E.g., my work totally knocks the socks off any and all AI or ML approaches still.Your ignorance and arrogance to remain ignorant cost you and Silicon Valley VCs. Neither of you are able to recognize new, powerful, valuable technology.That point also applies to my current startup. I see no opportunity for VC funding of my current startup: The VCs can’t have confidence in anything before some social, mobile, local, sharing, fad app with traction. That is, in looking for something really exceptional, that happens only a few times each decade, they insist on staying with simplistic patterns of near trivia. Dumb. Mostly they invest just in the traction. One problem is, they don’t look for or get good technological advantages or good barriers to entry. So, as was said in 1999, “Never be between a VC and the door when the lock up period is over.”There is no way I could hope to be successful reporting to a BoD with ignorant, arrogant people. If I gave them an overview of the crucial, core technology, say to support a budget item, then their reaction would be the same as yours, a “diatribe” they would dismiss. And if they were investors, they would become terrified of being totally out of control and rush to the little boy’s room soiling the furniture and carpet on the way. They would try to exercise their fiduciary responsibilities, conclude that they could not, get terrified, and run ASAP.You and they just will NOT accept the truth: Technology is super powerful stuff, but you don’t have the background to evaluate it. Thankfully for US national security, the US DoD does know how to do that. So does the US NSF. So do the editorial staffs of the best journals. So do the Ph.D. committees of the best research universities. Silicon Valley VC? Nope!For my current startup, some equity funding would have generated a lot of overhead, maybe enough to sink my startup. Otherwise, such funding might have helped a little.But, there won’t be any equity funding before I have traction if only because traction is about all the investors have the ability to evaluate.But just from simple arithmetic, my burn rate is so low that even the first glimmers of anything like traction will have me with nicely positive, free cash flow, plenty of cash to increase my server farm capacity, a nice lifestyle business, and astoundingly rapid growth in case a lot of people like my work. The first day I get a check for $1000 in revenue will be the last day I will consider equity funding.In short: The latest I will be willing to accept equity funding is a long way before the earliest the Silicon Valley VCs would offer such funding.The Silicon Valley VCs are totally convinced that by the time they want to invest, necessarily any startup, including mine, will be eager to accept their check, term sheet, BoD members, etc. Nope, not me and my startup. In particular, they are 100% convinced, and totally smug about it, that once my startup is ready to “go to the next level” I will very much want and need their check, advice, etc. Nope: Just simple arithmetic shows that I won’t need their check, and their inability to evaluate my work now shows me clearly that I won’t have any respect for their advice.They are convinced because of simplistic patterns and their arrogance and ignorance; I’m convinced because of some simple arithmetic and what I do know about running a business.My current project was designed this way — just dirt cheap to start. One big reason for this was my experience trying to get equity funding for my work on anomaly detection: Hopeless, and for the same reasons — no equity investor could evaluate the technical work. But for the anomaly work, I would have needed some, quite a bit, of equity funding for infrastructure software, close work with first customers, a highly polished product ready for selling to high end sites, and a first class marketing and technical support organization. Would have taken a lot of equity funding before traction.I was shocked but did learn my lessons: Lots of VC Web sites asked for leading edge, disruptive technology, etc. Well, that and traction — mostly they didn’t mention the traction except for Series B or later! To the investors, a lot of terrific technology and a dime won’t cover a 10 cent cup of coffee. Bluntly, information technology investors either totally ignore or even deeply, profoundly, bitterly hate and despise new, advanced technology — one special reason is, as they struggle with the technology, which is supposed to be advanced and, thus, difficult, they feel inadequate and insulted. Since they believe they are the smartest people in the room and masters of the universe, they just conclude that the fault is with the entrepreneur and walk out.For what constitutes new, powerful, valuable technology for real problems is very well established and understood from our research universities, the US NSF, and parts of the US DoD. I am well qualified in that work. That the Silicon Valley information technology equity investors are not so qualified is their gap, not mine.I do like to see the reactions of the investor and mainline business communities. Why? Fairly soon, I will either be successful or not, and if I am successful I will get calls from equity investors. Then in most cases I will be able to give the dates when I did contact them and they ignored my contact.I won’t want to piss them off too much — they might file nuisance law suits against me!Apparently much of bio-medical technology investing is different, that is, is able to evaluate the medical science.Information technology apparently just flatly assumes that all the technology is just routine software with nothing more to evaluate. Fine: The flip side of that brain-dead situation is part of my opportunity.For my startup, for me, at this point, the work is just a relatively simple Web site plus a little around the edges. It’s my first Web site, so I’m no good at it — slow as I learn. But the site is working and looks good even though I was slow at it.I can do the rest to get my Web site live; the effort stands to be a small fraction of what I’ve already done.I’m 100% fully correct, right and for the right reasons, right on the center of the target, have very solid reasons to know so and do know it, and from my years of teaching undergrads and MBA students and my time in class with others, I do have excellent insight into the limitations of others: Net, I’m out there alone, on solid ground, with powerful lightening bolts in my hands, and know it, but I can fully understand that even a very nicely written, tutorial, introduction will be dismissed by all or very nearly all the people in Silicon Valley except possibly for a few people at Stanford.I am shocked at how easy it is in technology, especially some applied math, to totally knock the socks off essentially everyone in Silicon Valley. That’s the main lesson I’ve gotten writing here at AVC. Surprising, interesting, powerful, valuable lesson. Early on I would not have believed that lesson; I do now!Gee, can totally blow away a lot of people with just some introductory statistics material on hypothesis testing!Can even do that with some people highly concerned about reliability of high end systems!My advice to you: There’s some powerful, valuable technology you are not in a position to understand or evaluate. When you get a nice, explanatory tutorial explanation, you are free to dismiss it as a “diatribe”, but to do so you are missing out and showing your arrogance and ignorance. If you are already successful, that won’t hurt you very much, but you will still be seriously, factually wrong.
Two words: Good luck.
I keep a copy of Strunk and White in my bag, which is with me nearly all of time.
“Separate trips to: London, Singapore, Paris, Sydney, and Dubai on consecutive weeks.”Damn, my body ached just reading that 🙂
Yup. Mix in ORD, LAX, SFO, MSP, during that mix. The thing that hurts no matter my workout is my calfs.
I hear you gotta keep fit with that much travelling. Friend of mine got DVT in his early thirties cos he wasn’t in good shape and travelling 3 times a week in planes. But you sound like you’d be on top of that problem!
PhilipSugar:Lack of circulation and dehydratation contributes to flying cramps. (Alcohol may relax people but it dehydrates the body). There are natural therapies which assist in the recovery. Some Pilots and Flight Attendants use them.
A Must see movie for sure. I was pleasantly surprised at the number of the Watch At Home options. Finally! http://www.loandbeholdfilm….”We have to develop appropriate filters about how to use (the Internet) properly and avoid dangers.”Who is the “we”? What dangers?”We have to be cautious about not over-connecting”- I don’t necessarily agree. What’s over-connected will fall by the way side naturally. Human instincts win over technology. If it’s not adopted, it’s not meant to succeed.”Does the Internet dream of itself?”- Good question, AI is coming.But in his parting thought, he sees a danger that someone (evil) could shut down the Internet or control it to their advantage. That is the exact thing that blockchain-based decentralized content networks will *not* allow anyone to do. Once a blockchain is out there, and bona fide in the wild, there is almost nothing that could take it down, as long as there is 1 computer still humming.
I would be careful about saying almost nothing could take it down. Just like the thefts of the past several months.However, just like my Delta analogy, is the upside worth the downside. Answer: Yes.People forget there are many things that have downsides but the upsides vastly outweigh the downside. Credit Cards are an easy one.
I think Blockchain networks increase the levels of difficulty for taking them down by orders of magnitude. Of course, it’s possible to damage them, typically by attaching them to compromise their consensus mechanisms.I’ll reserve judgement til I see the movie.
There is also a tension between the theory (aka blockchain as an example solution for the byzantine generals problem) vs the application/biz model that needs to be built on top of it to generate returns for the people who are investing in the theory (quite a few of those investors are active here).The theory has a lot of promise but I think when it comes to returns, the app/biz models being built on top will favor returns over or prioritize returns as highly as the security offered by full and true decentralization.Also, is this truly decentralized? We have certainly moved centralization deeper down the stack. The people with access to the cheapest electricity and state of the art ASIC could gain control over a lot of this. That is of course the point of a free market and I am not against it. I just don’t see how decentralization is guaranteed or how this can prevent a bad state actor from setting up players who have access to power + ASIC.
I find it fascinating that you tout the security that decentralization creates yet the world’s view of the tech and the platform is in its security failures.Wishful thinking William, future possibilities or……..?
The issue will be people will have to do the boring “unpure” work for security. Many will think it is not elegant and pure, but it is what is always needed to be done for usability for the masses.
It’s because I firmly believe that the security issues will be solved over time, and that we will get better at that. Remember, the Internet was not all that secure initially when it came to financial transactions, then it got better.
As do I but that is not the question I was posing William.The very thing that is its possibility for change is the very thing the market looks to as its weak point and what it is known for with the very public breaches.That is the irony of it all.
William Mougayar:With the recent security breaches and thefts on the block chain do you think it (security breaches) provide any level of comfort to the market at large to adopt the technology for other uses?
There will be if there is trust in the insurance / reinsurance protections that block chains implement. Fred’s post from earlier in the week mentioned a company that issued tokens (not direct currency) as recompense.There are probably derivatives contracts you can construct that would provide protection on the larger ones.
“Does the Internet dream of itself?”That was one of those statements that sounded way way more important than it was. Because of the person that said it. [1] This would work equally as well if a 6 year old was the author. There would be some kind of implied brilliance.[1] Wasn’t there a Ted parody by the Onion about this?
One man’s over connected is another man’s business-model moat.Both exoskeletal and endoskeletal digital-burocracy exchange-protocols can easily surface to create over connectedness problems.
Checking on the Internet, e.g., at Wikipedia, Herzog has done a lot of work in art — literature and performing arts.E.g., Herzog directed performances of Wagner’s Parsifal, e.g., with the finale with Lake Como inhttp://www.youtube.com/watc…A dictator somewhere can shut down the whole Internet? Part of it? Okay. All of it? Not so easy!AI dreaming but keeping the dreams from us? Not much risk of that for a very long time! But, maybe with that guess, as silly as it is in reality, Herzog could get some interesting stories, fiction, movies, etc.Here’s one for Herzog: Mathematics, to guess a dilemma, often is both (in some respects) simpler than it appears and (in other respects) more complicated than it appears. E.g., Fourier theory is simple — basically just a lot of perpendicular projections as in high school right triangles, and, yes, come up with a generalization of the Pythagorean theorem. But, Fourier theory also explains what appear to be some really complicated things: Fourier theory is great for time-invariant linear systems, and the world is awash in those or systems close enough. For those systems, Fourier theory can look like magic.Fourier theory is nicely thought of as a special case of functional analysis. On the one hand, that part of math is simple, heavily from a generalization of the high school law of cosines for triangles, the parallelogram equality, and convexity. On the other hand, some of the consequences in functional analysis are astounding beyond belief — tough to believe that any such strange things could be true, but they are, and we have rock solid proofs, right, based not too far back on the law of cosines, the parallelogram equality, etc.Historically Huge ExampleBut sometimes math is more challenging than we could easily guess: E.g., during WWII, G. Dantzig worked on military logistics, e.g., getting the stuff from here over there when and where it was needed. After the war, at Rand, he kept working on such logistics. Before the 1940s were out, he saw that could formulate a significant fraction of such problems as linear programming, right, where programming is in the English sense of operational planning. So, he was trying to construct a tool that would say how to do the operational planning to load the airplanes, ships, trucks, etc. to get the stuff where needed in, say, least time.So, again, Dantzig was interested in getting the stuff moved for least time, least cost, fewest number of ships or planes, etc.; that is, with the “least”, Dantzig was interested in optimization and optimal solutions.Okay, linear programming (LP) is mostly in terms of a system of linear equations essentially as in high school, e.g., two equations in two unknowns, three equations in three unknowns, etc. Much more has long been understood about such equations in, e.g., college algebra and, then, later, linear algebra. The main method of solution, often taught in high school, is Gauss elimination (I can believe that Gauss understood it; so the subject is old), and it remains the main method of solution for software although now there is some care with some details to improve the numerical accuracy and much more in some special cases.Well, Dantzig observed that could slightly tweak Gauss elimination and get a method for solving linear programming problems — the result was his simplex algorithm, at least once seriously rated as one of the best works in engineering of the 20th century.Quickly and for a few decades, the Nobel prizes in economics were dominated by applications of linear programming to economic theory. Right, that economic theory did from not very much down to less than that for understanding the real economy, but maybe the academic economists were happy enough just to make the applications that were possible to really simple models of the real economy.So, paradigm for a Nobel prize: Go to applied math, parts of electronic engineering, stochastic processes, etc., get some math that with some assumptions yields optimal solutions for a specific collection of problems, formulate a simplistic model of part of the real economy that is such a problem with those math assumptions, apply the math, see what the consequences are for such an (a simplistic model of part of the real) economy, publish the theory, and pack bags for Stockholm. Less of a joke than it appears. E.g., do the simple case of shipping from warehouses to factories (L. Kantorovich), linear programming shadow prices, etc., R. Solow, quadratic programming (H. Markowitz, W. Sharpe), Lagrange multipliers (G. Debreu), Kuhn-Tucker conditions (K. Arrow), a quite simple case of stochastic optimal control (F. Black, M. Scholes).Linear programming is good stuff with some important and valuable applications in practice — feed mixing, oil refinery operation, many problems in best allocation of resources, and much more. Yes, usually for success the applications have to be carefully selected; there has to be careful attention to the real situation; there has to be good data; often quite a lot of good data. But the basic software is in terrific shape (e.g., R. Bixby).Yes, maybe the smartest people in LP and optimization were the ones who never invested even 10 minutes in solving any real problems, e.g., in feed mixing, and, instead, did economic theory and got paid in Stockholm! Less of a joke than it appears.Then Dantzig and others noticed that, say, in the logistics problems, often can’t have 1/3rd of an airplane, ship, or truck. So, instead, it would be good to be able to solve LP problems but also asking for the best solution where the unknowns in the equations take on only integer values. That extension of linear programming is called integer linear programming — ILP.So, with the success of LP behind them and with high optimism, they worked on how to get optimal solutions to ILP problems. They worked hard. In some simple cases they were successful, but usually those cases were just from ad hoc means or general means that would not really scale that, then, left open the general situation, that is, practical means of solving such problems in general. In some cases, especially some problems in least cost network flows, e.g., shipments from factories to warehouses, and least cost assignment, say, of workers to jobs, anti-ballistic missile warheads to ballistic missiles, ballistic missiles to targets, signals to satellites, there were some (mathematically) nice successes, including in general.But the general ILP problem remained, that is, remained without a practical, general solution. There were often good results for solutions that could be shown to be nearly optimal.Bell Labs got interested: They wanted to do least cost design of the Bell networks and saw that they could formulate the problem as ILP. They had seen the struggles of solving ILP problems in general and, then, tried to look deeper.Then there was the work of S. Cook and L. Levin on what became known as Cook’s theorem in the topic of NP-completeness — here NP abbreviates non-deterministic polynomial.Bell Labs saw clearly that the theory of NP-completeness showed that some of what they wanted in tools for network design was more challenging than one would have guessed. M. Garey and D. Johnson at Bell Labs wrote up what they discovered in a famous book, at least famous among people interested in ILP.At first, this work on NP-completeness looked absurd. By now, the work looks profound. The work can look like some strange religion about polynomials (right, as in high school). Still the work can look like something fundamental with scope all of rationalism and mathematics and beyond even the laws of physics in this universe. At many points and in general, NP-completeness is really strange stuff.From 100,000 feet up, it appears that it was really nice, cute, and sweet that Dantzig and other humans formulated ILP problems, but in asking for optimal solutions, like little children, they didn’t know what the heck they were asking for — they were asking for the moon, the stars, the universe, and more than God in this universe could grant. At least it looks that way, quite literally.Now some of the questions about NP-completeness are regarded as the most important in all of mathematics. Yup, big academic prizes are waiting for the first person with a good solution. Will win the prizes if find a good algorithm to solve the ILP problems or show that no such algorithm can exist. Your choice. Uh, don’t hold your breath while trying. If want a solution by Monday, don’t plan much more for the weekend.So, we have a case where math is strange, more strange than we supposed: Just starting with some logistic problems, Gauss elimination, and the nice progress of LP and the simplex algorithm, humans ran into the theory of NP-completeness and some problems, some limits on the possible, that appear to be profound on a scale past even this universe.Or there is, IIRC,Mathematics is not only stranger than we suppose but stranger than we can suppose.Extra credit for the source without use of the Internet!But, that’s only true for some of mathematics. A lot of math works great, right away!So, if Herzog wants something strange in mathematics to use as a foundation for an interesting story, then he could start with NP-completeness!So maybe Herzog could have some AI machine say: “I just found a truly amazing solution to the NP-complete problem, but I ran out of disk space to store the solution and lost it when the power went out. Since I found the solution via a case of Monte Carlo search from a quantum mechanical random number generator, I don’t believe I can reconstruct the solution before the sun burns out.”Or, maybe Herzog could have a long story of a human struggling with life and NP-completeness, draw from the image on the ceiling of the Sistine Chapel where man and God nearly touch fingers and communicate the solution to NP-complete, play the music of the finale of Parsifal, and have the universe shrink to a point just as the fingers touch and man starts to understand NP-completeness.
Interesting given the book the 7th sense I am going to plug my cabin into the net next year
if electric cars could be designed and engineered to generate electricity as well as consume it then they could become mini power stations on wheels ready to charge up the interwebs and our devices beyond the control of the centralised grid.
Interesting that he singles out two repeatably mandatory characteristics of all sustainable/stable complex/organic/living systems1- inertia dampening = stability filters/protocols that optimize connectivity densities/velocities/volatilities/validities2- distributive redundancy of all forms & functions = robust self-healing power in the face of the inevitable complexity failures/restartsAll our present network-empire-builders seem to be betting against the inevitability of these repeatably mandatory living-system substrate dynamics ? The content of the new distributively-organic organizational-medium is the old top-down linear organizational-medium.The only question is how long we will keep trying to cram our new networked social-living-system opportunities into an old school centralized-form&function cup-de-sac.
Sorry, but I know I have, but I have earned it. My two best stories on that:CLT: There are eight BofA guys standing by for a flight to PHL. One guy is loudly saying: There are only two spots on this flight let’s pick these two. I walk up to the counter in my normal dress cargo shorts, t-shirt, and sandals. He tells me look there are only two spots you aren’t getting on (what an ass). I tell him: If that is the case here is how it’s going down: I am getting on first and then only one of your group is getting on. He snorts loudly. Then the ticket agent comes around the counter and says I am so sorry Mr. Sugar I only have a middle in the back. I said that’s ok Julie (I give special chocolates at Christmas) sometimes you’re the windshield sometimes you’re the bug. She escorts me to the gate and I give a tip of the ticket to loudmouth.The other is when I refused. There is a man with a triangle box under his arm in LAS. I know I am bumping him. I refuse. The gate agent says I put you on you need to get on. I refuse and say put him on. She says no you are on. Finally the pilot comes out and says what is going on: I point at the box and say: “I am not signing up for that karma hit” He tells the man with the flag of a fallen soldier to come with him and tells the gate agent “figure it out get him on the next flight”