A Decentralized Archive

The Internet Archive and Wayback Machine are awesome.

You can see that in 1998, AVC.com was pointing to something called Fishnet that looked like this. And you can see what AVC looked like in September 2008.

There are many archives, some of them quite specialized, in the physical and digital world. But they all suffer from the same problem, which my colleague Dani explained yesterday:

Archives are extremely important: the survival and ongoing availability of knowledge helps with the continuation of human progress. While there are existing digital archival projects out there, they are mostly donation-based and face the risk that their funding could run out, their hosting provider could one day go out of business or a government could force them to remove files.

Dani wrote that yesterday in our blog post welcoming Arweave to the USV portfolio. Arweave is a decentralized protocol that enables archiving and the funding to support it at web scale.

The key innovation is a sustainable funding model for archives which Dani explained in her post:

Arweave had to invent a new method of paying for storage, one where you can pay once, and store forever. While that sounds almost too good to be true and took us a long time to wrap our heads around, it will be true provided the cost of storage continues to decline at a predictable rate. Machines that provide storage to the network get paid out in small increments over time as they continue to prove that they have held onto archived files. 

Decentralized infrastructure is one of our favorite investment themes in crypto (Filecoin, Helium, Arweave, and at least one unannounced investment) because of the resiliency and sustainability of decentralized infrastructure (just look at bitcoin mining for an example of that).

We think infrastructure will tend to move to decentralized models over time and archives are a great example of that.


Comments (Archived):

  1. awaldstein

    intrigued but a bit challenged to get my head around this model and how it works.I don’t think I can spreadsheet this out. If someone has, super interested.

    1. Michael Brill

      An oversimplified version… assume all-in storage costs drop 50% a year [1.0, 0.5, 0.25, 0.125, …]. Then your in-perpetuity storage costs approach year1costs * 2. So if you want to store content permanently, you pay 2x the current one-year market rate. But since they have a conservative cost reduction model, it’s likely going to be *way* more than 2x (10x?) and all goes to hell when the value of the AR token tanks relative to the long-term fiat-based operating costs.Also, I don’t understand why they model their pricing on HDD pricing when that’s maybe not even a material component of the total operating cost for a node.

      1. awaldstein

        Thanks.In concept i get this.I just can’t model it from what i see.And if I can’t understand it, i’m hesitant to pay today for a distant future result as i doubt we it will be there is all.Since everything is getting more expensive in the real world of things–like storage–i want this to be the counterbalance.

        1. Michael Brill

          They just need to move it along far enough so the $5m investment can turn into a $50mm Reg A+ offering. That window is probably open for another 18 months.And the day of reckoning for immutability is coming anyway.

          1. awaldstein

            ‘just’ ?

          2. Michael Brill

            Sorry, I’m off the charts cynical about institutional investment in crypto projects these days when liquidity is realistically only going to come from retail investors who are going to lose most of their money. It’s quite a different dynamic from reliance on public markets or institutions for exits and, to me, tarnishes the heroic narrative of venture investors that I’ve kind of grown to like.

          3. awaldstein

            all good.i have one pos project but all others are in games with a conservation funding mechanism, a regenerative ag market and considering one in the digital art exchange space.love these categories actually.

          4. Michael Brill

            To be sure, there are tons of projects that I love, although I’m biased towards the tokenization/fractionalization/composability aspects that have immediate(ish) value. I simply don’t understand how making things 10-100X more difficult and grinding a most technical innovation to a halt through decentralization of services makes sense. And there are the practical issues like whether this sort of immutability is going to survive things like GDPR and law enforcement (it won’t). And who in the world wants to pay money in order to give up their right to delete their own content?However, there is 20% of my brain that acknowledges that the people working on and investing in these types of projects are way smarter than me. I might just be that guy who doesn’t understand why anything than a 3270 terminal is useful. :-/

          1. awaldstein

            Thanks.I’ll consider reading as I’m interested.But whenever i come across something interesting, important that the only way to pursue is by digging into massive primary links I consider it a failure of communications and storytelling.Not a question of value or intent. Communications is 100% of the value of an idea and of implementation.

  2. John Pepper

    I just want to say thank you for bringing back our old web sites of the late 90’s and early 2000’s… I always regretted that nobody had saved them in some manner. They are all there. Amazing to go back 20 years in an instant.

  3. William Mougayar

    “Proof of x” using cryptography and immutable records to secure data is a very good use case. But curious why is a special token needed for that model to work? Users could pay for it in fiat, stablecoins, bitcoin or Eth, no?

    1. Michael Brill

      The volatility of ETH and BTC would be a bad match for long-term fiat-based costs. Also, who would invest in this if there wasn’t an opportunity to dump a new token on the unsuspecting public through Reg A or an IEO?

    2. Lasse Clausen

      Why does Bitcoin have BTC, or Ethereum ETH?

      1. William Mougayar

        Not sure I’m following the question.

  4. RiganoESQ

    decentralized storage is most important part of the new internet. thank u

  5. kidmercury

    great innovation, i setup a wallet and look forward to testing it out. that you have to setup a wallet and download keys will limit mainstream user adoption substantially in my opinion, but i’m sure the passionate early adopters are already there.i don’t think these types of dapps, in which a currency is tossed out into the wild and the market (in this case storage buyers and storage sellers) is expected to regulate itself with it, is viable without centralization. eventually calamity will strike by design (competitive attack) or by the natural course of events, and a centralized entity will need to emerge to stabilize the token in a manner that ensures survival of the system (in this case the storage archive). those who think bitcoin is decentralized and hold it up as a model of anarchic decentralization should remember the binance bitcoin hack, in which binance lost 40 million USD worth of bitcoin to hackers — and actually floated the idea, to the dismay of many bitcoin early adopters/purists, of essentially bribing miners to reorganize the chain so as to steal back the stolen coins. binance eventually just took the loss, but as wealth flows into crypto and stakes get higher, eventually such reorgs will occur. it is better to just embrace the creation of governance systems (which inherently will be centralized) that allow for reorgs to occur, along with more easily enabling additional benefits like custodian wallets that facilitate onboarding.

    1. Michael Brill

      All that is to say that real decentralization doesn’t (and probably shouldn’t) really exist?

      1. kidmercury

        i woudl say decentralization and centralization are a continuum, in which one side can never be 100% dominant

        1. Michael Brill

          But so many of these projects are predicated on complete decentralization and simply fall apart once you expose even a tiny bit of centralization (which of course they all have).

          1. Mike

            My take, this will be a centralized business using blockchain techniques to offer a decentralized storage service. I would be hesitant to view centralized/de-centralized as a binary choice.

  6. OurielOhayon

    i was excited at first but found it to be really hard to use and dropped it. did i miss something?also i fail to understand how the “forever” claim is garanteed. unless it means forever as long as arweave somehow exists?

    1. kidmercury

      i read their yellow paper quickly, what i got was that they take a part of the money that emerges from the one time fee for the addition of a new item to the archive and use part of it to reward ongoing storage. how they allocate what is for the initial addition of the new archived item versus is what is used for storage is based on some mathematical model they have which assumes perpetually falling prices of storage, a trend they cite has been in place when viewing the price of storage in US dollars (though i would suggest that it is quite critical to note that storage has not been perpetually falling when priced in other currencies, i.e. argentine peso, iraqi dinar, etc).if i misunderstood and anyone can correct me i would very much appreciate it.

      1. jason wright

        Something like that. A Moore’s law for tokens?

        1. Michael Brill

          Yes, although in a p2p model, the cost of storage is essentially zero as it’s an unused asset (that extra 100gb on your drive). All your real costs are power, network, compute – so why base the magical token economy model on storage costs?If it’s going to be more of a client/server model then it’s going to be highly centralized anyway as a small number of entities can operate huge storage farms at the lowest possible cost.

      2. OurielOhayon

        i do understand the economical logic but i do not understand how the promise of “forever” can be garanteed and enforced should the incentive mechanism fail

        1. kidmercury

          yes, though i don’t see it explicitly stated, i’m assuming the belief is that the enforcement mechanism will never fail.

          1. OurielOhayon

            the lack of clarity is worrying. add to that the problem of user experience i had (maybe this is for developers only). i don t know…

          2. kidmercury

            it seems to me that it is really for developers at this point, or at least those comfortable with managing their own keys. bitcoin is still early adopter-ish, and the most popular ways to obtain and use bitcoin remain through centralized exchanges that abstract key management for users (coinbase, binance). so IMHO i don’t think end users are ready for this type of decentralization but rather will want something that offers more guarantees at the trade-off of more centralization, like what coinbase and binance are already doing with much success.

          3. OurielOhayon

            you are probably right although i am 100% comfortable with managing private keys, as this is what i deal with

          4. Lasse Clausen

            There is no “forever” claim. Estimated to be 200-400 years.Also, Aweave is the protocol that is among the easiest to interact with.

          5. OurielOhayon

            how is the 200/400 years estimated. more importantly guaranteed?

          6. Lasse Clausen

            At some point you’re gonna have to put in the work and read their documentation 😉

          7. OurielOhayon

            i did. but i do not understand except that i have to trust and believe their incentive mechanism will sustain

  7. jason wright

    I read the ARW (what does that mean?) post yesterday. Now it’s vanished. Hopefully ARW archived it before it disappeared, because if not it was never a thing at all.What really caught my eye was a reference in the post to some sort of democratic voting process to decide what does and what does not get archived on the ARW network. Is this for real?! Imagine if Internet Archive had functioned in this way. Fishnet would probably never have been recorded.Soren Kierkegaard’s The Crowd is Untruth comes to mind. I assume the AR founders have read it and disagree.

    1. sigmaalgebra

      One summer I was force fed Soren Kierkegaard. Now I have a lot of sympathy for the geese of Strasbourg.And maybe Kierkegaard was handing out what he had picked up from what the geese had dropped! Even not force fed, an experienced farm housewife remarked about geese “They are worse than pigs.”.From memory it was I and Thou? Naw: I just looked it up: I and Thou was from Martin Buber. That was some more of the force feeding that summer.For me, tough to tell apart since it looked and smelled all the same. I didn’t want to step in it; I sure didn’t want to study it.I was a math major and nearly a physics major. The college had a really good ugrad math department but also that summer force feeding Strasbourg program.

      1. jason wright

        Fear and Trembling, to correct politically correct tendencies 🙂

  8. Mike

    Very clever. I can see the linkages to the Helium model. Why not sell the local storage you are not using, managed by the blockchain? I am sure centralized storage services are taking note.The mining instructions seem very “early adopter”. It reminds me of HTML coding for web sites in the late 90s. Quite a lot of evolution yet for mainstream markets and users.