Data, Transparency, and Regulation

Last month, I pointed to a talk that Nick Grossman gave at Princeton where he laid out the principals of Regulation 2.0. This slide is from that talk.

Regulation 2

Regulation 2.0 is a framework that we have been working on with a bunch of others who are rethinking what government means in a networked world. In the Regulation 1.0 world (the one we are in now) regulators are required to give you permission to do things. In a Regulation 2.0 world, as long as you report openly and transparently about what you are doing to the government and everyone else, you are free to innovate and operate. But you are accountable to live up to the rules that are set by the regulators and the data you report about your actions will be measured against those rules.

I thought about all of that when I read about Mayor Bloomberg's team of data crunchers in today's New York Times. This group, which is described as "the half-dozen post-collegiate techies", operate with a budget of less than $1mm a year and yet have been able to solve many tricky problems for city hall in the past three years.

This story is a good example:

Last fall, the city’s Department of Environmental Protection wanted, finally, to crack down on restaurants that were illegally dumping cooking oil into sewers in their neighborhoods

The antiquated answer would have been to have the health department send inspectors to restaurants on blocks with backed-up sewers and hope by chance to catch a busboy pouring the contents of a deep fryer into the street.

number-crunchers working from a pair of cluttered cubicles across from City Hall in the Municipal Building dug up data from the Business Integrity Commission, an obscure city agency that among other tasks certifies that all local restaurants have a carting service to haul away their grease. With a few quick calculations, comparing restaurants that did not have a carter with geo-spatial data on the sewers, the team was able to hand inspectors a list of statistically likely suspects.

The result: a 95 percent success rate in tracking down the dumpers. With nothing grander than public data, the Case of the Grease-Clogged Sewers was solved.

This story reminds me so much of the story Steven Johnson told in the Ghost Map and many of the stories he tells in his current book, Future Perfect. It should not be a surprise that Steven is one of the folks who have been working with us on this Regulation 2.0 framework.

As more and more of the data about what goes on in our world becomes available via network organizing structures, we will be able to regulate much more lightly, thus lowering the cost, and burden, of government and allowing innovation to prosper.

New York City, under Mayor Bloomberg, has been leading the way in using data and technology to locate and address problems in an efficient manner. But there is so much more that can be done in this direction. I hope that his efforts are adopted and evolved by his successor and all governments in the coming years.

#NYC#regulation 2.0

Comments (Archived):

  1. leigh

    My 2 biggest beefs with big data:1. People think the data is the strategy. To me the genius is always how to use it as a strategic tool without it becoming the strategy. It’s easy for people to confuse the two.2. People get all excited about the data they can get, when they don’t even action the data they already have (e-commerce sales data for a start….)ps. fav recent article on it was found via AVC i think — Netflix & big data

    1. William Mougayar

      Yup. I attended an e-metrics conference this week in Toronto where big data was often part of the discussion.This summarized my take-away:Reporting data is dumb. Analyzing data is smart. Be creative while being analytical. Know what the data delivers.

      1. leigh

        Key summary is a great quote πŸ™‚

      2. JamesHRH

        Need to know the business issues to analyze the data – these people are not thick on the ground.

      3. CJ

        “Reporting data is dumb. Analyzing data is smart. Be creative while being analytical. Know what the data delivers.” – So many gems in this threat today.

      4. awaldstein

        Measurements have never been interesting. Interpretation is everything.Big data or a little data–still the same.

        1. William Mougayar


    2. PhilipSugar

      When you use it as an excuse not to think about the underlying issue is my problem.Use big data to analyze what customers are saying about you.Here is an idea. Get better customer service.Alfred Lin, CFO of Zappos had a great insight. He refused to put in systems to collect and analyze how long customer service people took to resolve a call because he realized that would prevent them from providing good service.

      1. LE

        Agree. I tend to see things like this on more of an analog level by what makes sense rather than trying to see if there is a problem when the numbers indicate a trend or a problem with the numbers and then take action (insert obvious “when it might be to late”).I was in Panera this morning to pick up coffee and a bagel. The line was “to the door” (took 20 minutes to get through) and there was no organization to the line at all. It was difficult to see where to stand and I inevitably stood in the wrong place and had to go to the end. There was no line control at all. This is typical in an otherwise very well run store that has great products.A person relying solely on numbers would not know there was a problem (or know of lost opportunity [1] from frustrated customers) until it showed up on a spreadsheet. But it was obvious to me just standing there that they needed this problem solved because it’s going to make me think twice about going in there on Sunday mornings so they will lose my business perhaps simply because they aren’t using line markers which are easily obtainable and inexpensive. To me (as someone who actually ran my own retail location in the business I was in) this is something that I would have noticed immediately (I even had cctv installed at the front counter and hung around frequently to see what was going on so it’s not difficult at all to monitor activity. )[1] Lost opportunity meaning you can be making money but there is something you could be doing better that allows you to make even more money or build a better business.

      2. CJ

        “Alfred Lin, CFO of Zappos had a great insight. He refused to put in systems to collect and analyze how long customer service people took to resolve a call because he realized that would prevent them from providing good service.” – Do you have a source for this Phil? We’re revamping our internal CS experience and I think this would be a good point to bring up that counters the move away from phone and towards email.

          1. CJ


          2. PhilipSugar

            Buy the book.

          3. CJ

            The book?

          4. PhilipSugar

            Referenced in the article: Tony Hsieh’s book, Delivering Happiness: A Path to Profits, Passion and Purpose.You can get one for free if you take the free tour of their call center in Las Vegas.

          5. CJ

            Cool, I’m there in May so I’ll do just that.

        1. PhilipSugar

          I actually had a long chat with him on this. I am not opposed to email customer service. The key with email is getting back really quickly.

          1. CJ

            I’m not opposed to email CS either, but I don’t want our userbase to lose that feeling of personal service. A voice on the end of the phone builds relationships quicker than an email trail, I think,

          2. PhilipSugar

            In my experience it gets the person asking for support to think about and articulate the issue. If you reply back quickly you also get the benefit of being able to provide the answer in writing where they can reference it. Also nothing prevents the support person from calling.Now if you use email to try and automate and get to them when you feel like it?? Bad.We provide either choice, no cost, it is included. We only ask people to try email once. More than 9 out of 10 go to email. I’ve never understood the it takes two days. Either way you have to get back to them, so add people. I guess the hope is you get abandons but if that is your goal, just go like Google where you know you aren’t going to get a reply.

          3. CJ

            Our userbase is full of dinosaurs. LOL They wouldn’t touch a PC if they didn’t have to and they believe in the power of the phone call. It’s how they conduct 95% of their business with the outside world so they expect for that to carry-over internally.That said though, I’ve seen the spark of recognition that occurs when you speak with someone on the phone for an hour and meet them in person vs. exchanging emails. It’s almost as if you know each other already and that leads to better internal satisfaction metrics. Fast is good, but a connection is better. At least in my experience. I want both and that’s the line we’re trying to straddle.

          4. LE

            What is the average price of the product that you sell?

          5. awaldstein

            Depends on what’s the goal.Ex–had some serious Mac issues over the last week, some hardware, some software.After one call into AppleCare support they gave me a direct line to a brilliant guy here in NYC who with two calls revolved stuff.Support was better that the product….is the product I should say.Aggravated by the issues, enthused by the care!

          6. PhilipSugar

            No don’t get me wrong. I am opposed to email only customer service. Apple takes it a step further by letting you actually see somebody in person with an appointment so you don’t have to wait.

          7. awaldstein

            Yup…genius bar is a bit of genius.Direct dial customer support routed to people in your time zone is just so smart.They get that when your computer breaks your world suffers and waiting is never good thing.

          8. LE

            “The key with email is getting back really quickly.”Even if just to say “we’re working on it” and not in an autoresponder type fashion either most importantly.The fear of the unknown is what causes anxiety and aggravation in all situations.That said it’s also important to pace email responses. If you get an email and respond really quickly then points will be taken going forward when inevitably you aren’t able to keep up that reply time consistently. Contrast principle.So if it’s reasonable to be able to consistently reply within 2 hours (arbitrary to make a point not saying that is right or wrong it could be 15 minutes [1]) then you should reply in 2 hours and not 5 minutes, 10 minutes and then 2 hours even if you can. Otherwise people will reply back with another question and expect to get a response as quickly or be disappointed if they don’t.[1] Response time is totally dependent on the product, service, price and a host of other factors. All I’m advocating is to be consistent given the particulars. You will note that most likely you can’t write an attorney and get a reply back in 5 minutes as it would be to interuptive for someone like that to have to maintain that as a work flow on a consistent basis.”The key with email is getting back really quickly.”Second point on this though. I’ve had the opportunity to do business with some people who are the type that are really bad with email and replies. You can be discussing something and it can take days for them to reply back to something you are discussing. Or even longer. Not because they are super busy or traveling but just because they are bad with email or the amount of times they check or how fast they type or how important it is to them. In that case for me in particular it disrupts my work flow to deal like that because I have to hold in memory something that I can dispose of really quickly if there is a back and forth that is closer to real time.

    3. LE

      1. People think the data is the strategy. To me the genius is always how to use it as a strategic tool without it becoming the strategy. It’s easy for people to confuse the two.I couldn’t agree more. We put a man on the moon with computers that had nominal power because of effort and determination to do so and in spite of lack of resources that are available today.The data tools are great but simply saying (as the NYT did) that “The antiquated answer would have been to have the health department send inspectors to restaurants on blocks with backed-up sewers” ignores the fact that if someone with the desire and a clue had decided to get involved in solving this problem without shiny tools it could also have been done. [2][1] Back when there were only video stores with videos I always wondered why nobody had a wall or two of videos arranged in a “like this then you might like that” (where “this” might be “godfather” and “that” might be “goodfellas” similar to what Netflix does today.) Guess what it doesn’t take big data to do that it could easily be done by any number of people who have seen a fair amount of films.[2] NYC health department ratings in windows as an example of something that essentially takes very little technology instead relying on simply a good idea and human nature to solve a problem.

      1. leigh

        Clients always want us to gather more data — I always ask why. Just in case is often the answer. Not a good enough answer so we only track data that we intend to action or that will contribute to meaningful analysis

        1. PeterisP

          Trends are a big issue – if you find out that data point X is valuable and actionable, then you want it’s history for at least a full year (due to seasonality), preferably more. If you have been throwing that data away for the last five years… then it hurts your ability to analyze and predict. It tends to be cheaper to just store and organize all sources of metrics just in case.

        2. ShanaC

          part of the problem is you may need other data later

    4. JamesHRH

      Media moniker making always causes problems when you try to communicate these ideas.Bloomberg does not have ‘number crunchers’ doing this work – he has investigative analysts.’Big Data’ is not the new horizon here (large organizations have had more data than they know what to do with for decades) – unsiloed, accessible data is the new horizon.Last week, I asked 2 utilities in Calgary to provide me with the last 6 months of statements. 1 had them in my inbox while I was on the phone with the agent; I am still waiting on the other one (3 to 5 business days to get an email out to me.They both had the data……but only one had systems that made it accessible.

      1. ShanaC

        i’m actually surprised more essential services aren’t forced into giving over data (eg: banks and your statements)

        1. JamesHRH

          I live in a border city now – the idea that a custom officer cannot pull your VISA transactions to check for duty & taxes……seems hilarious.

    5. ShanaC

      3) not even knowing what data to look at – so you develop a GIGO problem

  2. takingpitches

    As you know, I am a 1000% kindred spirit in less permission regulation leading to more innovation, so this is just a corollary to your post which I agree with.The one place — directly connected to having more data making us smarter — I think we need more effective regulation is privacy.Society has until the digital age been premised on a certain degree of anonymity to live your life, the ability to ultimately keep certain things private, or have them fade from memory where that has failed.The main response to this is personal responsibility, and realizing the reality, but as private companies have access to more and more of our deepest thoughts and secrets through providing us email or hosting our cloud documents, we also need protection that such data is not abused, or we will be very sorry later.As Nietzche observed: WIthout forgetting it is quite impossible to live at all.

    1. Aaron Klein

      Or…we could just not use services we don’t trust with our data.We need to be better as startups in providing a simple visual summary of terms of services so that users can build trust. But government trying to regulate how Internet services can innovate is a recipe for disaster.

      1. takingpitches

        maybe, but hard for many folks to not use gmail, for example, and privacy policies for such services are constantly changing. hard to know who to trust and opting out of using services is not really an alternative. i think some, smart regulation is necessary, but do realize the paradox in that very statement.

        1. Aaron Klein

          Google is not and should not be under any obligation to provide me with Gmail under terms of my choosing.

  3. pointsnfigures

    My biggest beef is if government uses big data to be big brother. Bloomberg has done some good things-and then some things that are horrible, like banning Big Gulp’s and other regulations.I don’t like the notion of “nudge” when used by governments. Your example is a great use of data. However, there are uses that could become pernicious in the wrong hands.

    1. PhilipSugar

      I agree 100%Where I live you would never have to worry about this because people pay to get the grease. They actually lock it up because you can turn it easily into bio-diesel to run your truck or boat. http://www.popularmechanics…. Its worth money. BTW: the leftover is used to make those homemade soaps that cost big bucks.In NYC I’m sure to protect a constituant that paid off a public official you need a carting license and it actually costs money to get something removed that’s worth $4 gallon!

      1. LE

        I was going to say that as well re: bio-diesel I wonder what the story is in NYC with the dumping.

        1. PhilipSugar

          29 total licensed companies to pick it up. As you put it that is a barrier to entry, one that I’m sure comes with paying somebody off.

      2. LE

        “you need a carting license”I know someone that bought a junk franchise (similar to 1-800-gotjunk) in Pennsylvania. I asked him if he would open one in NJ. He said in NJ it takes something like 8 mos. to a year to get approved for junk hauling because of organized crime, the casinos or something that sounded like that.So he didn’t want to go down that route (he wanted something easy with no effort.).I saw it differently more of an opportunity to open a business where there was significant friction to entering in the state so it was well worth the trouble! Had I been interested in this particular business that would have been a plus not a minus.

    2. Elia Freedman

      Except I believe this is how it should work. Dialing in the right levels is really hard. Bloomberg went a little too far for people, they fought back and re-calibrated the correct equilibrium. Someone pushes, someone pulls, and over time we find the right comfort zone.

    3. fredwilson

      it would probably have been better to levy a heavy excise tax on “big gulps” to provide to funds to care for all the diabetes patients they create.

      1. PhilipSugar

        If people had to directly pay for their medication instead of having the government pay for it, I think many would try and lose weight instead of taking insulin and drugs.

        1. fredwilson


      2. pointsnfigures

        The problem is using government as the solution to the problem. Collecting a tax to …….something else never is a zero sum game. Multiplier effect of government spending is 0. There is always leakage. Of course, it is impossible to isolate issues-since the entire food chain in health care is screwed up and only getting worse with Obamacare.

        1. kidmercury


      3. kidmercury

        Who says big gulps create diabetes? Some would argue diabetes is a mineral deficiency. Maybe tomorrow someone will have a different viewpoint supported by the scientific method.

    4. kidmercury

      What government wishes to track, government wishes to control.

  4. takingpitches

    p.s. I love Bloomberg’s effort, and think it’s important to figure out who is dumping grease and where ambulances should be stationed, but even more interesting would be an office of a digital advocate whose job it would be to advocate for entrepreneurs in front of city agencies that are currently designed (or have been captured) to protect incumbents, i.e. help hailo and ubers and others as they dealt with the TLC, help AirBnB deal with the regulations that they may be tripping over as folks rent out their apartments.This would be truly radical, but perhaps it inevitably must be a private function.

    1. LE

      “office of a digital advocate”How about an ombudsman for poor people that get shafted by the system and don’t have the ability to solve even simple problems without involving an attorney?

      1. takingpitches


    2. fredwilson

      to some extent we are doing that right now along with a select number of other tech folks in NYC

      1. PhilipSugar

        One you could start with is BIC which has only licensed 29 licensed grease haulers in all of NYC that number is ridiculous, there are more than that in small counties.

    3. ShanaC

      isn’t this the role of the chief digital officer in part? and the public advocate?

  5. Kevin Prentiss

    This slide is equally applicable to management at large companies and marketing at every company. We are speeding up, 1.0 process can’t keep up.

  6. Anne McCrossan

    Fred thanks for recounting this and kudos for your work. I see this as a great example of organic value generation and quantified organisation – using data to help organisations as they are now, yes, and also using data to radically re-conceive, potentially, the entire way we handle the act, and the art, of organisation. That excites me.

  7. William Mougayar

    This was a great quote at the end of the NY Times article “…government is a hackable system β€” an operating system that can be optimized.”Governments are sitting on a ton of data whose potential shouldn’t be limited to them. This goes beyond regulation, and into innovation.Yes, in some cases, it can be used to regulate/enforce like the example of the grease dumping. But that’s only scratching the surface. I think the greater innovations will come from the private sector using government data and combining it with other data sets to predict new things, generate new insights, develop new products and services, etc.Data science is definitely going to take a leap forward with all this open availability of new sets of data. It’s the new R&D.

  8. JLM

    .Sometimes we outthink ourselves and try to create uses for technology — those gee whiz moments — where a simple bit of enforcement would be easier, less expensive, require no infrastructure.Every restaurant requires an annual, on site health and food safety inspection including the functionality of the grease trap.Why not simply inspect the grease cartage permit and contract at the same time?Too simple?JLM.

    1. William Mougayar

      Yes, but they would have had to send inspectors to all restaurants on those city blocks where there was sewer blockage. In this case, the data helped them to single out the potential culprits. Probably, it was also less costly to the city.But to your point, it makes you wonder if all restaurants are regularly inspected or if they only do spot checks.

      1. JLM

        .Circular argument.They already had the data indicating who did not have a cartage permit right?They waited to act until they had a problem. Typical enforcement inefficiency and laziness. The best solution is always systemic prevention.Every restaurant in NYC gets an annual inspection. They have to be posted.JLM.

        1. PhilipSugar

          This is not an issue where you live JLM. Go into a restaurant and ask them to take their grease. You will have to pay them.

          1. JLM

            .But the real issue that Fred raises is one of the consequences of not disposing of grease correctly which is typically an issue with the sizing of the grease trap.Modern grease traps which seek to contain 100% of the grease have a problem with their efficiency and efficacy — they do not work when overloaded.When a grease trap gets overloaded, then the grease just enters the sewer system.The prospect of being able to sell the excess grease which simply means that the same function is now a revenue generator rather than an expense is really universal.In one instance, the restaurateur is getting some $$$ and in the other instance the grease trap servicer is getting the $$$. The grease is worth money in any zip code.JLM.

          2. PhilipSugar

            Except in NYC where it costs money to get it taken away!!! Not only that you have your choice of only 29….that right in Fred’s link, 29 by law are the only ones that can service 17,000 restaurants!So the cynic in me says if I want to shake down restaurant owners how do I do it?Well since I know I can find a grease trap violation in any restaurant, and if I send somebody out and they don’t find a violation that reflects poorly on them, I tell my 29 cartage guys, look I’ll do an enforcement on anybody that’s not using you, that will protect and increase your monopoly, I’ll make it a big deal, put it in the press so you can really stick it to those bastards.So now instead of having to deal with hundreds and hundreds of inspectors and tens of thousands of restaurants and worry somebody films one or somebody gets divorced, finds God etc. All I have to do is tell 29 guys: ok I want a share in the Biodiesel that you sell. Because that’s hard as hell to track and you can sell it for cash, Voila!!Suddenly a politician that makes $100k a year amasses a $10mm net worth which happened to my Senator.

    2. LE

      “Why not simply inspect the grease cartage permit and contract at the same time?”They could actually do this by issuing a contract to a third party to do the work. Then create an incentive system whereby the inspectors are paid to write tickets for infractions instead of overlook them. With a chance to get a re inspection and obvious controls. Inspectors get rewards for finding things and reporting even if the situation is corrected later. The “inspected” get to fix the problem and not be fined. Or something like that (give me more than a minute and I’ll think of a more airtight solution).I’ve had plenty of inspectors show up over the years at various places of business. My typical response is not to allow them in when they arrive and simply get them to reschedule at a future date giving time to correct anything obvious they might find. Being nice about it of course otherwise they would tend to be less lenient.

      1. JLM

        .This goes to the systemic issue — why pay more for the enforcement mechanism that is already in place? And already paid for?If there is a cost savings by privatizing, then it makes sense.JLM.

        1. LE

          Have yee not had thine coffee yet this morning? [2]Because it wouldn’t cost more setting it up outsourced rather than with government employees it would cost less as they would be more motivated and efficient and could have their assess fired for not doing their job. Instead of being sent to rubber rooms. [1] [3] [4][1] Assuming of course you could get it by the unions.[2] Or is it me who is over cafinated?[3] Don’t forget when setting this up to rotate the inspectors so they don’t do the same area otherwise they will get to friendly with the prisoners and cut them breaks in return for personal favors.[4] I held onto the business card of the local fire inspector kid who is a fireman as well. I needed the serial number off the roof HVAC and told him I would pay him $50 to climb up on the roof and simply take a picture and get me the serial number. He replied back as follows:Sir, Unfortunately this is outside the scope of employment . I apologize but I am unable to help you with this matter. My advice would be to contact the property management team for further assistance.

      2. PhilipSugar

        That solution sucks.

        1. LE

          Reason? Which part?

    3. fredwilson

      that’s old school, wasteful, expensive, subject to corruption, and not going to be the way its done within a decade or two, hopefully soonerhow many building inspectors have you had to pay off in your day JLM?i find the current regulatory model offensive

      1. PhilipSugar

        I think you are arguing against yourself here. Petty corruption is always going to be a problem. But where you get big time real money corruption is when you get private enterprise to charge an inflated price to business owners who have no choice because of laws. In this case you can then get real money from those enterprises because it is merely a cost of goods sold. Think trash concrete or here grease.So in all of NYC I as a restaurant owner have 29 total companies to choose to get a cartage permit. You can’t give it to Phil’s service where he uses the bio-diesel to power his trucks and the glyercin to make soap.Reminds me of when I owned EnviroMetrics Software. The EPA was so busy trying to analyse the data that we gave them from Fortune 100 Process companies, they didn’t go out there and enforce. Sure we’d massage those numbers something fierce but who really cared when you knew your customer had a massive benzene spill they didn’t report.I would love to see software replace the administration of government. I don’t mind paying for officers, firefighters, teachers, inspectors, its the others that I mind.

      2. ShanaC

        you should look into the idea of tav ha’yosher. They’ve effectively become the department of labor voluntarily on both the enforcement side and the consumer side for kosher restaurants…

  9. PhilipSugar

    My worry is when people use big data instead of looking at the underlying problem. They lock grease up where I live because it is worth money.Maybe if NYC didn’t have an obscure agency that protected both the people carting away the grease and added costs to the restaurants it would be like most of the U.S. where the grease is worth more than a dollar a gallon.Even LA is better….look these guys got arrested for taking the grease! http://latimesblogs.latimes

  10. Dan Cornish

    Bloomberg is an angel, so he would never do anything bad with the data he collects. Everyone who worries about the government over reaching and using data to punish and nudge is wrong. I know throughout history the default state of government is tyranny, but this time it is different.

    1. Jim Ritchie

      I love the smell of sarcasm on Sunday morning…it smells like tyranny. Pan to shot of Bloomberg’s private drones blowing up a 7-11 full of Big Gulps with Wagner’s Ride of the Valkyries blaring in the background…Seriously though, what do people thing about Bloomberg’s latest comments on use of domestic drones?



        1. fredwilson


          1. FAKE GRIMLOCK


          2. ShanaC

            why do you say that?

          3. JamesHRH

            No hiding behind fun online persona.

        2. ShanaC

          i wonder if that will be legal. 7th (?) circuit already decided that the documents on a computer are covered under the 4th amendments, inclduing encrypted documents. Why wouldn’t the feed be covered

        3. Aaron Klein

          Yes, but wait until your Google Glasses are equipped with remote-controlled mini-Hellfire missiles. THAT will be badass.

    2. PhilipSugar

      I would love to see him use the big data to analyse the cost of that obscure department, how much the cartage fees cost residents, and how many bribes those employees take.

      1. ShanaC

        that would be amazing, and unlikely

      2. Aaron Klein


    3. CJ

      The data is already there, might as well utilize it.

      1. Dan Cornish

        Here is some more data the angels in government will get to use. “Starting in 2014, per the dictates of the federal government, your doctor must record your body mass index (BMI), which measures whether you are overweight, each time he or she treats you and turn it over to the government via your electronic health record, which every patient is required to have. Your BMI will then be tracked by the Health and Human Services Department, the agency rolling out ObamaCare, and a bevy of other state and federal agencies.Read more: http://www.americanthinker….

        1. Dan

          This is a bit of a misscharacterization. Providers need to collect “vitals” including blood pressure and bmi for 50% of the patients they see if they want to receive funding from the government to use electronic medical records. There is no requirement for all Americans to have an ehr and this data is not reported back at an individual level.

          1. Dan Cornish

            It starts with 50%, but Doctors will collect 100% so they do not get in trouble and to make book keeping easier. This is one example of the “nudge” philosophy. The politicians can “claim” it is not required, but we all know the government will get data on everyone. And once they have the data, and since they are paying the bills, they can require anything they want “for the common good”

    4. pointsnfigures

      Whoa, I respectfully disagree. In the Wall Street Journal today, Gordon Krovits writes about the advantages of using Big Data-but also says, “China shows the dark side of big data, monitoring everything from web usage to cell phone locations in order to block protests and arrest dissidents.” The Bureau of Homeland Security in the US is doing similar things with Big Data. That’s wrong. As we apply it to more and more things, government’s use of data could eventually lead to less freedom, not more.

    5. kidmercury

      Thinking about this stuff gets in the way of creating and taking easy stimukus money.

    6. ShanaC

      But not every pol is. Definitely the article is right about freedoms being overstepped

  11. rich caccappolo

    Enjoyed the article – I’ve heard of several really interesting cases addressed by the office of policy and strategic planning – they deserve a lot of credit. Interetingly, just before I read this article I had read the editorial also in today’s paper “Imprisoned by Innovation” in which the author argues “Technological innovation does not guarantee political innovation; at times, it might even impede it.”. He makes the point that certain uses of smart technology deal with symptoms, not the underlying problems, eg, “reminders to eat more vegetables don’t help if you live somewhere where they are not available” and “self-tracking can get us to optimize our behavior within the constraints of an existing system. What we need is a chance to reform the system itself β€” perhaps by dismantling those constraints. So, back to the case of the grease dumpers – along with catching those who were breaking the law, is there a solution to the underlying problem of disposing of that waste? I bet there is and let’s hope that the move to regulation 2.0 drives such innovation, as listed in the slide.

  12. William Mougayar

    One of the key things about analyzing data is to really nail down the Correlation vs. Causation aspects. If the data leads you to the “cause” of something, that’s the real insight…and it’s the hardest part to figure out. Typically, correlation is much easier to spot, and you can get easily lost in it, e.g. the relationship between increased sales of ice creams and drowning, but one doesn’t cause the other.Correlation and Causation applies very much to startup metrics.



      1. William Mougayar

        Yup. Big Data = Big Trouble.Startups don’t need big data to figure things out for themselves.

  13. RudyC

    I think everyone on this post today has one thing in common. Complete and utter distrust of any POLITICIAN. I have always admired you Fred for your vision and mostly because you seem straight up, but in this one you are completely wrong.Data is used for more than one thing and only one thing when it comes to gov’t, manipulation. While I can recall many a times I think the most telling example besides the unemployment rate, would be the crime rate.The murder rate is this country has decreased by 50% over the last 20 years, yet no one says anything nor takes credit for this. Why? Fear and the machine. A decrease would me a decrease in funding which in turn would mean a decrease in law enforcement.Sorry but data is manipulation of the people period. Maybe if the populace was better educated but in this day and age, that isn’t the case.

    1. William Mougayar

      Maybe that was the past. With richer and more open data, I’d like to hope that both the government and private sector have an opportunity to go beyond that.

      1. RudyC

        @william…maybe in canada…but this is the good ole USA…sorry but the big guy ALWAYS screws the little EVERYTIME!

    2. PhilipSugar

      I remember discrediting a keynote speaker at a conference. He said he said the big data said people were better entertained and didn’t go out and commit crime.I asked him how about the increase in the manditory sentencing and the increase in the prison population:…He started gasping like a fish out of water.

      1. RudyC

        I tried looking for the stats briefly. As a wise man once told me, they’re always a multitude of reasons for something ending in a given result, though honestly I don’t think stricter sentencing has much to do with it. The point though is that gov’t manipulates data to serve a purpose. I betcha Iraq could possibly fall into this category as well..

    3. fredwilson

      not everyone on this post today has one thing in common. i disagree with you completely

      1. JamesHRH


    4. ShanaC

      i would data is the manipulation of an imperfect representation of a person. I am not my data body. I’m similar, but not the same.

  14. WA

    Bravo big data.

  15. Elia Freedman

    I’ve wondered for a while whether transparency is the better answer to campaign finance. Let people donate what they want to whomever they want but every person who donates goes on a publicly available website for anyone to see to whom, how much and when, all the way down to $5 Internet donations, all sortable and filter able, even at the state and local level.



      1. CJ

        sad but true

      2. ShanaC

        it is why “fitting for culture” can be so rough

    2. fredwilson

      i think we already have that. maybe i am mistaken. you can see all my political donations on the web at various services like and

    3. RudyC

      I don’t know…let’s ask Obama!

    4. LE

      Fred isn’t looking to hide anything, to wit on that sites he referenced [1]:”Union Square Ventures/Founder”I would imagine it would be fairly trivial though (especially with a name like “Fred Wilson”) to use other valid info that would not make it obvious the money that he is contributing. I mean what is to prevent Fred’s daughter from contributing as “student” or Fred to contribute under another business name? Such as his real estate holding company. Sure that could be uncovered but it’s not exactly low hanging fruit. I’ve also seen people will deliberately misspell their name in an “oops sorry for the typo” move as well.[1] Of course for completeness I will mention that that could also be a red-herring.

  16. LaVonne Reimer

    Strangely I woke up this morning thinking of my own blog. The idea was “outside the black box” a subject very near and dear to my heart. That’s the prism through which I see big data. In the black box or out. Expert-driven or user-driven. I applaud this use of data in NYC but what stopped me was this: “number-crunchers working from a pair of cluttered cubicles . . . dug up data.” My question isn’t whether that’s good or bad. It’s whether that’s really the essence of Regulation 2.0. I was imagining something that looked more like the power of the networks to create self-regulating frameworks.

    1. LaVonne Reimer

      Well not strange I was thinking about the blog but that I woke up with thoughts of black box models which is a near waste of a great sunny morning.

  17. YoniAssia

    This is a great initiative, regulation is indeed a great challenge in any startup trying to disrupt in financial services. There is no doubt the values presented for Regulation 2.0 are more aligned with the social web, and the era of information networks, the question for me is how do we get from 1.0 to 2.0 . Like in the startup world, it seems that many times the jump from 1.0 to 2.0, does not happen through the incumbents adopting new technology, it happens through new and innovative companies that don’t need to “fight” all the policies and procedures already in place in the 1.0 companies. In regulation that would suggest we should give up on the existing regulators and simply build new ones, that would require a lot of courage and political power, but would probably be the quickest painless way to create regulation 2.0.At least in financial services,maybe we should support a NEW self regulatory organization lead by VC’s that would be based on regulation 2.0, and only accepts companies which provide full access to all their data.Think about reinventing investing in the network era, where innovative and agile thought rule, new transparent capital markets, new social investment networks, accountable investors, VC’s that invest billion in the new public markets, real time KPI’s in addition to quarterly financial statements, everything as API’s, everything global, everything open, everything on the internet. Everything settled in bitcoin πŸ™‚

    1. fredwilson


    2. LaVonne Reimer

      If we were to take a communitarian view of natural law theory (my bias as informed by Mennonite heritage and a long-ago Jurisprudence professor who was a student of self-regulation in Native American cultures), then wouldn’t self-regulation be more effective and natural to have companies collecting and sharing data with VC’s on a level playing field? VC’s in turn would use these interactions to further inform companies on evaluation criteria. In effect, self-regulation would be manifest through collective analyses. Or something like that!

    3. ShanaC

      when have you heard of new and innovative governments though? And when have you heard of old regulators giving up power?

      1. YoniAssia

        Only when no one is asking them πŸ™‚

        1. ShanaC

          Meh. Who really wants to give up power…only happens when a new better power is exchanged for it



    1. MikeSchinkel

      Sounds to me like a perfect reason to study Jevon’s Paradox.



    1. fredwilson

      i would rather report on what i am doing in real time than have to be visited by the dept of building inspectors twice a year and pay them off to sign off on my building

      1. LE

        “i would rather report on what i am doing in real time than have to be visited by the dept of building inspectors”But the purpose of an inspection is to inspect in person what was done and that it was done correctly and according to code. In the case of fire inspections to test the alarm system, access and entry, emergency lighting etc.



  20. Dave Pinsen

    Technology usually isn’t the limiting factor here.A few years ago, I saw a news story about someone (a physics professor?) who invented a gadget that could tell which cars were polluting as they drove by, by zapping their exhaust with a laser or something. The report speculated about how that could obviate the need for state inspection stations. That hasn’t happened.

  21. Richard

    Greatest Potential for Big Data = F (Data Volume, Data Velocity, Data Variety, and Data Veracity**)**Lets not forget about the errors embedded in big data.

  22. Sean Saulsbury

    If “freedom” isn’t the first word in regulation 2.0, there’s something wrong with it.

  23. MikeSchinkel

    @fredwilson:disqus As I read this a part of me thought “This is absolutely awesome!” Then the other part of me thought of Pfc. Bradley Manning. πŸ™

  24. opoeian

    Brilliant concept and well explained Fred. Inspirational, yet again!!

  25. kidmercury

    This is bound to lead for fighting for stimulus dollars. A fool’s endeavor.The real solution is in New governments and new cities. There is a city in Georgia, USA that is a new city — just incorporated — that is all network-ish. That is the startup that needs disruptive technologies, not old school wall st money like new york that is already an incumbent depend3nt upon an entrenched value network.The larger issue thoufh is the assumption that the regukstor has the moral authoeity to continue in that role. The viewpoint put forth in this blog post also disregards the internstional environment which is “the big picture” that will ultimately drive all significant changes in the regulatory framework.

    1. MikeSchinkel

      What’s the city Kid? (I live in GA.)

  26. jason wright


  27. Steven Kane

    can i agree with everything you wrote but humbly submit that this post misses the biggest issue?regulation is not only about policing — finding and punishing people after they commit offenses — which is what is illustrated by the (wonderful) item you cite from the nytimes piece on bloomberg administration data, i think the most important function of regulation is *prevention*.I think thats true whether its the dumping of pollutants in common waterways (the problem cited in your post) or the type of widespread fraud and criminal activities that led to the financial system meltdown in 2008.i’m wildly energetically hopefully in favor of transparency, accountability and innovation (is anyone sane really against these things?), but i am also deeply terrified of trying to create a deregulated world that looks to market forces to police itself — or to generate the data needed to police after the fact.i mean, heck, will we never learn? will we really come out of the mind-blowing global disaster of 2008 only to revert to the well-meaning but utterly failed ideas that nearly ruined the liberal democratic capitalist world? (and that still might.)

    1. fredwilson

      i am not talking about a de-regulated worldi am talking about regulating it differently using real time data

      1. Steven Kane

        huh. well i’m not sure thats the way the post is read.when i see “Permission” listed on the “regulation 1.0” list, i assume that is meant to indicate its an old or bad or obsolete idea. don’t you think most people see it that way?and what is “Permission” if not regulation itself? regulation that stops people from doing things without “permission”?likewise, when I see “Trust” listed as the bridge to the new paradigm, i can’t help but assume that is a euphemism for deregulation. or less regulation. or a lever to be used to argue against regulation.i mean, isn’t the assumption, who needs regulation when there is “trust”? isn’t that what the slide means when it uses the word “Trust”?what can I say? I am very uncomfortable trying to use emotional constructs and concepts like “Trust” when working to create the legal architecture for creating a common good.Such efforts end badly — Trust me!;)

  28. Prokofy

    I’m having trouble getting this story, Fred.On the one hand, you want to have businesses “free to innovate,” so you sort of have them obey regulations on the honour system, correct?So instead of intrusive inspections for everyone, or instead of lengthy applications for permissions, you just have people “trusted”.On the other hand, you want to consent to have the government intrusively monitor everything you’re doing (i.e. what you dump down the drain) and then on a statistical basis, pounce on you and prosecute you.So in exchange for not applying for a permit, you’re willing to give the government the go ahead to monitor and scan you endlessly, thoroughly — for whatever.Are you sure that’s a good idea?