We have an open source database company in our portfolio. The company is called 10gen and the database is called MongoDB. This is a fairly techie subject and the pros and cons of data stores is not a topic I broach very often on this blog.
But MongoDB is worth talking about. As 10gen's marketing says, MongoDB contains "The best features of document databases, key/values stores, and RDBMSes in one".
MongoDB was built by some of the former engineers who scaled Doubleclick to be the largest ad server in the world. In the process they found out that traditional relational databases presented some hard scaling challenges and the experience left them asking if there was a better way. MongoDB is their "better way".
And now MongoDB is available as a hosted web service. So if you want to play around a bit with MongoDB and see if it's a good choice for you next web project, just go to MongoHQ and request a beta invite.
If you do that, I'd love to hear what you think. Either in the comments or via email if you prefer.
Hi Fred,We use S3 for our document storage and the Key Value Pair architecture really works for us. Is MongoHQ primarily going up against S3?I am curious to see MongoHQ differs from S3?A key suggestion to differentiate MongoHQ from S3 is to provide value add on the document/file itself. For e.g. If its a word/pdf document, can one automatically convert it into a flash player embedded document so that the web service doesn’t have to do the additional work of converting and showing it on the website. In other words, for every document stored in the DB, can MongoDB provide an easy web representation of the document so that its not just a regular DB but it actually works extra hard to make your documents/files web “compatible”.So, if the the NYTimes, has documents, videos, files etc, you not only store the files for them in a scalable fashion but you also do the additional work for it to easily show it on the web.Maybe, a stretch analogy could be a Twilio for document/file storage.Nik
That sounds pretty nifty Nik. How do you like using S3 so far? How simple to get into was it, compared to something like Googles datastore?
Actually, its quite to use S3. We didn’t try Google’s data store. But we did try and see if we should install our document storage DB (some of the players are mentioned here- http://www.metabrew.com/art… ) but ultimately decided to go with S3 because of simplicity, familiarity and scalability…But, our documents don’t live in the ether, the documents we store (like quite a few other web services) need to be shown on our service and “hacking” a web compatible format for all the various document formats is not fun…
Without knowing much of anything, I believe that MongoDB, especially a hosted version, is much more like the combination of Amazon S3 and Amazon SimpleDB (both the database-like features as well as the actual large data store)…S3 is really just about data storage, while SimpleDB is about giving you some database-like features from a key/value system…Awesome stuff though on all fronts for us techies to learn and play with! 😉
I need to familiarize myself with setting up a nice scalable database. I’d rather use a system that handles most of the difficult work of keeping it fast no matter how much it bloats.
Most all of these key/value type databases (Mongo, SimpleDB, Google’s Big table) are designed to solve just that pain point…right now the real question in picking one over another seems to be what company/environment you want to be tied to (ie. Google, Amazon, or your own hosting world)…and this directly relates to all the other tech. decisions you’ll be making (like what language you want to work in, what your tech. budget long term is, what type of scale you expect to have to support, etc.)…
Appreciate your shared experience in this area Kevin.
I shall explore their database API, but my limited expertise in this area will be a quick judge (if it’s too tricky I’m skipping it). I want ultra simple kindergarten building blocks for my web apps 🙂
that is good feedback
I wasn’t able to sign up, seems I need a special code? edit: nevermind, found the request beta code link
I’d say it’s more analogous to SimpleDB than S3 — it’s a hosted database. Although it is possible to store very large objects/files in MongoDB so I suppose it is an alternative to both.It’s easy to run MongoDB on EC2 also, so that is an option too if one uses it. (Of course one could connect to a mongohq instance from ec2 over the internet too.)
We’re really interesting in MongoDB but have been waiting for the auto sharding capabilities to be ready for production. Have these limitations been overcome in the hosted version from 10gen?
Fred I really think I want to talk about this… ; ) …a challenger to S3 without the pressure of retail going through Amazon…yeah I really want to talk about this.
How do S3 (or any of the amazon web services) put pressure on retail going through Amazon? They are backend platforms that have nothing to do with retail, and you could easily build a website using s3, sqs, simpledb, and ec2, but use paypal for your payments layer instead of amazon payments. You could even build your own internet bookstore on AWS if you wanted to…What I like about the 10gen stuff is that you can use their sdk to develop scalable software, but potentially deploy it anywhere, whether on 10gen’s cloud, some other hosting provider, or on your own hardware.At a glance Mongo looks like a CouchDB / a hybrid of SimpleDB and S3, but I haven’t needed to really dig in to any of them yet.
We are a different entity. …that Amazon would like to see our retail/referral go through them.
I don’t understand what you’re saying, but all of amazon’s services aredecoupled: you can pick and choose and use any or all of them.Here’s an example of a 3rd party service that someone wrote to let you selldigital goods using S3 as your document store, and paypal as your paymentgateway. Amazon doesn’t discourage this, even though it could be doneentirely on their platform. In fact, it is one of the featured services inthe AWS showcase.http://www.digitalgoodsstor…
of course we can decouple it. We are a different situation. sorry
I’d have concerns about the overhead of every db hit from my app-server going over the public net.
Yes – going over the public net would be an issue. I think the best use case is in the same data center as the hosting service. Eventually, MongoHQ should have servers at the big hosting companies, so you just use the servers that are local to you.
I think there is a huge trend for “on-demand” services, related to hosting. Of course S3 and EC2 set the trend, but we now how MongoHQ for databases, we have Superfeedr for RSS feed parsing on demand… etc.
I discovered mongodb in a post you did here back in february and instantly fell in love with the concept. I’ve been using it in an under development app and i’m very pleased with the features and performance. Their mailing list (a google group) is very active and the founders/employees promptly answers all sorts of questions, and judging by myself and all the excitement i see there by people discovering mongo, i think you did a very nice investment on this one that will definitely pay off :)I do host mongodb on my own dedicated server, but i think that the hosted offer on mongoqh is an excelent idea as many developers don’t want/don’t know/can’t afford a dedicated hosted and usually go with a shared hosting where mysql is usually their only database option.
that is great feedback. thanks so much.
So all this talk of databases is above my head. And I think that’s silly when it comes to the market of databases.I was talking to family friends very early this summer, at the first mention of the whole NoSQL movement. Not that I fully understand it. This family friend had used SQL and was looking for a change in life, she was in her late 40s-early 50s, and was looking to say try to learn SAP in some mysterious way. NoSQL hadn’t even hit her radar, and it still really isn’t, despite my insistence that it will.One of the reasons is that the literature if you are just starting out is way above most people’s heads. Aim to explain everything. At the most basic level. You’ll capture a market no one else has yet (I mean, why would anyone else go for that market, we’re just not that cool…)
Cool- I finally get to put on my East Agile hat on this blog without sounding pitchy :)East Agile is an elite engineering team doing contract web and mobile development. We’d be happy to work with such an offering and off-load the DB heavy lifting that’s not really our core competency– this comes up more and more as we work with clients’ more robust and more spikey apps.Hosted MongoDB’s value prop seems similar to that of Engine Yard, who is probably the best solution out there for Ruby deployment and management (we are an EY development partner).As it relates to startups’ adoption, key to this is pricing model: Engine Yard just changed their pricing for smaller companies so the cost can start small and scale up if and when usage grows.(We’ll let you know how our Mongo usage goes.)http://eastagile.com
please do Ken
We really like mongodb, and actually use in a key piece of our infra. However, I’ll be extremely surprised if the hosted offering will be used for anything serious. It’s not only the networking latencies, cost and indeterministic performance. The main issue is that working in an architecture that is outsourced horizontally is extremely hard. Counter-intuitively as mongodb becomes more mature and robust, it becomes easier to deploy it and thus the incentive to use it in a saas model goes down.What will work, however, is to create something like amazon images that provide value and are vertical specific, like cloudera is doing successfuly. a simple example will be a great geo based db that provides a complete location awareness solution.Incidentally, one of the cool features of MongoDB is that it’s open source. For example, today we found a bug in mongodb’s handling of long vars somewhere by being able to dive into the code. Will be hard doing that in a saas model 🙂
I think this is a great way to demonstrate the simplicity and power of Mongo. I was giving sessions at a JoomlaDay event in NYC on Monday, and when I described Mongo I got a lot of quizzical stares…I whipped out my laptop and showed them the Mongo Shell, and suddenly lightbulbs were going off everywhere in the room.This hosted service is going to be a great way for developers that are unsure of the value proposition of Mongo to kick the tires and see for themselves what Mongo can do.
Exactly. That’s why I blogged about it.
Hi Fred,We’ve been fairly involved with 10gen and the latest builds of MongoDB. It has been a key part of our elastic infrastructure since we launched a few weeks ago, fitting perfectly between the other providers that we use (namely MySQL and S3). Scaling smoothly, Silentale currently stores more than 8 million large items (“documents” for MongoDB) and grows by about 4% per day. The 10gen team has been absolutely fantastic in supporting us and moving forward with their roadmap. I really hope that the hosted version will stimulate more companies to use and contribute to MongoDB in production systems.Laurent
Wow. Thanks for sharing this. It is so great to hear stories like this
I’ve been using MongoDB for a few projects recently. It’s an incredible system for rapid web development and has proved very solid for serious production work. We’ve even managed to sneak it into a deployment for the local arm of a large Japanese automaker.When I heard about MongoHQ I also signed up for the beta. As expected being an early release, a few rough edges but could be a great option in connection with other EC2 cloud solutions. As a test we’ve deployed an early prototype of a new app we’re building. Very smooth going, if they can nail it with the price then it will be very appealing to go alongside services like Heroku or plain old EC2.
That is great to hear hunter. We are hearing more and more good things about mongo
That’s great to hear. By any chance could you blog about your usage of the product? We are getting a lot of questions about how people are using mongodb in the real world.
We’re doing a webinar on MongoDB on Oct 30. It’ll be an overview of MongoDB, and will also have Ian White from Business Insider talking about how they are using MongoDB in production. This should answer some of the questions people posted here on MongoDB vs other databases, understanding production use cases etc.Details & registration at: http://mongodb1.eventbrite….If you have ideas for future webinars, which could be to share an interesting way in which you are using MongoDB or to learn more about a specific aspect of MongoDB, we’d love to hear. Please share your thoughts in the comments or email us at [email protected].