About a decade ago, I tried to index the web via the Dewey Decimal System. I had a site laid out similar to Google, where you could browse sites continuously starting from a given call, but the DDS is proprietary, and those people hate anyone who uses their IP without a license. You can Google everyone they’ve shutdown – places that weren’t even libraries – for using anything similar to the DDS. I reached out to the group that manages the DDS, and was taken offline before my project even started.
With all the corporate BS lately, and people looking for alternative options, I thought I’d take a search engine old school, and we’d index like Usenet. Except with an XXX.XXX.XXX format.
My original project used sharded Redis, with append logs to disk. I chose it for its in-memory speed and key-value store. Did some calculations, and I’d have to have millions of records just to consume my entire system’s memory.
I had a lot of plans for this before getting shut down.
Now that I’m older, I’m curious if I should be using MongoDB.
What are the benefits and drawbacks of each? Which would you use? And why?
Postgres is usually the DB to use unless you have a specific reason to use something else. Something like Cassandra or scylla might be good for your use case but I’d probably still start with postgres and evaluate a switch to something else if and when you need to
Mongodb is proprietary.
Postgres is open source, and then there is documentdb which adds a mongodb compatible layer on top.
But it would probably be better just to use postgres + jsonb directly for that case.
Redis is apparently open-source again, now under AGPL (and two other proprietary licenses). But yeah, can also recommend the fork that spawned when they went proprietary: https://en.wikipedia.org/wiki/Valkey
When did Mongo go proprietary?
Edit: I’m actually wanting to make my entire project open source. MongoDB’s current license, requiring all infrastructure to be open, might actually be for me.
No, the license is not a real OSS license.
The short version is that the requirement of all infrastructure being open, could potentially apply to parts of the stack where you don’t have access to the source code, like the management engine or network card firmware.
It hasn’t been tested in court yet, and because this is unclear, the SSPL is a hot potato nobody wants to touch or test. The possibility of the requirement for closed source firmware to be open, makes the SSPL effectively unusable.
The SSPL is neither Open Source Initiative nor Free Software Foundation approved: https://en.wikipedia.org/wiki/Server_Side_Public_License
Currently, SSPL licensed code is best treated as source available, but proprietary code.
Redis is not a database. If you use it like one, it won’t be long before you regret it.
Can I ask why? Are there errors in the append log? Does it forget things?
Not questioning your character. Just curious what your personal experiences were.
Technically, it shares a lot of characteristics with databases. You can put data in there and get data out again later, sure. But it’s optimized for short term caching of data, to reduce load on a real database, reduce network requests, that kind of thing.
In programming you can often use tools in ways they were not intended for and it’ll still work but the costs of that decision might not become apparent until much later. This is one of those. Over time you’ll start to have issues with durability guarantees, complex querying and indexing, transactions and consistency, recovery after failure, backups and restoration, data growth, operational tooling, migrations and schema evolution, concurrency, debugging and observability. None of that shows up during initial development work so you won’t notice until you’re in way too deep.
