Rendered at 23:04:45 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dullcrisp 1 days ago [-]
This article doesn’t present much motivation for why you would do this.
To me it reads a bit like, how we built our office without desks: it turns out if you stack two chairs on top of each other, you can balance your laptop on the top and you’ll also have a shelf on the bottom for your things.
tdeck 1 days ago [-]
> it turns out if you stack two chairs on top of each other, you can balance your laptop on the top and you’ll also have a shelf on the bottom for your things.
But that sounds kinda unstable and doesn't really provide the full benefits we expect from a...
Oh.
locknitpicker 1 days ago [-]
> This article doesn’t present much motivation for why you would do this.
For those working with databases, it's easy to come up with a bunch of reasons. ORM is a necessary evil when working with RDBMS, as are things like schema migrations that can easily result in loss of data (I.e., dropping columns or tables).
What if all we need is dumping a big old JSON in a container?
This idea is very enticing. The scale of this whole NoSQL thing is pretty telling.
The blog presents a thought provoking question: what if we don't actually need a full-blown database, and instead we only care about things like unique constraints, transactions, indices, and history tables.
What if we only need a subset of those?
embedding-shape 1 days ago [-]
> ORM is a necessary evil when working with RDBMS, as are things like schema migrations that can easily result in loss of data (I.e., dropping columns or tables).
Hmm, neither of those been necessary for me. ORMs is a choice, you can choose not to and still have a proper design and architecture and not suffer from that choice. Requires you have programmers who know how to use SQL, which seems less and less common as the days go by though, so understandable most reach for an ORM.
Schema migrations that can "easily result" in data loss is even easier to avoid though. Don't drop the column/table in the same step you copy data to the new place, or do the migration via application code which gets cleaned up later, to catch most of the stuff before one bigger copy, and then the eventually DB cleanup.
Guess it depends on how careful you want to be, or how careful your agent lets you be, I guess. But it's definitely possible to avoid both of those "necessary evil", which I guess makes them just "evil".
e12e 1 days ago [-]
Object databases exist - they don't magically solve data(model|schema) migrations.
> The blog presents a thought provoking question: what if we don't actually need a full-blown database, and instead we only care about things like unique constraints, transactions, indices, and history tables.
This sound like: what if we dont actually need a table and instead we only care about four legs and a wooden plate.
ryanbrunner 1 days ago [-]
Articles like this remind me of that Innovation Tokens article.
Your time and attention is precious as a developer. I'm absolutely sure it's possible to implement uniqueness constraints, transactions, indices, and history yourself, but is that really the most valuable use of your time? There's probably not a need for you to have a unique solution, so you're quite literally just re-inventing something someone already had for not a lot of benefit.
Wouldn't your time be better spent actually solving the problems that whatever you're building is supposed to solve?
bob1029 1 days ago [-]
Development of video games is the final boss of this one.
If you want to kill someone's dream of publishing a game, encourage them to build their own engine from scratch. I cannot think of a more malicious piece of advice given how effective it is, statistically speaking.
The exceptions to this are so incredibly rare. Virtually all of the in-house engine work at indie scale has been replaced by Godot in recent years. It used to be something like 10-11% of studios were in-house fully custom. Now it's probably closer to 1-2% fully custom and 8-10% on Godot. Having a reasonably stable OSS option has removed the last major argument that I am familiar with for simply using what already exists.
my-huge-pony 1 days ago [-]
>last major argument that I am familiar with for simply using what already exists
I was interested in gamedev as a kid. I always went with a custom engine. You're right, I never released/finished a large game.
But now I think I was just into game engine dev. Every time I finished my game engine and started working on game content I ingredient got bored and went to another project.
So I guess this is the reason, engine development is fun :).
jauntywundrkind 1 days ago [-]
You don't write your own libc to talk to the filesystem, do you?
Same vibe here imo. Use zerofs, juicefs. Use picomq, automq, s3stream. Use slatedb. Use lancedb, fusion, tonbo, iceberg, vertex. Use duckdb, data fusion, polars. These systems all natively speak to object storage. Use Qwikwit, rising wave / hummock, greptine mito, use warp stream, neon. Use celld. Over half of the projects here are very specifically about object storage.
Many deal with uniqueness, transactions, indicies. Which is extra impressive because its across machines. Managing data on a box creates a huge array of challenges and difficulties, backup and scaling and HA, etc etc. A huge reason to face object storage head on is that often you are headed towards that fray one way or another, and object storage makes an excellent easy to scale and manage disaggregation layer that keeps your problems from complecting together, is a clear, well known separation point. With an incredible ecosystem around it.
arpinum 1 days ago [-]
This is frustrating to read. Tigres is built on FoundationDB, but doesn't expose all FoundationDB operations like transactions, range reads, and get mapped range. They go through all sorts of complications to handle these issues, including a database for caching (and they don't consider thundering herd problems).
What if you just ran FoundationDB instead?
embedding-shape 1 days ago [-]
> What if you just ran FoundationDB instead?
Would that let you do collaboration blog-posts with another VC-funded startup though?
kjs3 1 days ago [-]
Priorities...won't people think of the really important priorities. :-)
ovaistariq 1 days ago [-]
Tigris founder here. So part of the reason we do not expose other database like APIs is because we are a S3-compatible store, so anything we expose has to fit into the S3 API semantics.
In our current design we use FDB as our metadata store which is the sort of workload it is really good at.
arpinum 1 days ago [-]
My complaint is about your customer's choices, not yours. The customer solved their problem, but they also created the problem in the first place by building on an abstraction that hides capabilities they need.
xena 1 days ago [-]
I've occasionally pitched the idea of exposing transactions for object storage based on bucket forking. I think the idea is finally starting to land, but it's a really "out there" usecase that is likely hard to conceptualize until it's usable in some form.
ddorian43 1 days ago [-]
FoundationDB is only as metadata store.
Also, the database is the last thing you want to re-invent unless your business is explicitly building a db (even then it's best to re-use a db like all the postgresql forks that have existed)
arpinum 1 days ago [-]
> the database is the last thing you want to re-invent
good news, FoundationDB is already invented, and just slightly more tested than others on the market.
agentultra 1 days ago [-]
> In practice, when you reach for a database engine you're actually reaching for four basic features: unique constraints, transactions, indices, and history tables.
That might be a very specific assumption. What about serializability? Replication? Materialized views? Procedures? Locking? Access control?
It’s cool to experiment and try new approaches. Neat one here.
Could still end up moving to Postgres.
thom 1 days ago [-]
What about... queries?
dam_jackalopes 1 days ago [-]
[dead]
i2km 1 days ago [-]
Along with putting estimated reading times (21 minutes in this case) can authors please start putting estimated writing times?
Did it take 10 seconds of prompting? Or hours of thought, trial error and revision? Especially when asking people to read for 20+ minutes...
ahknight 1 days ago [-]
They should put a Summary section at the top with the raw prompt.
tantalor 1 days ago [-]
> Protobuf field names are forever
If you only have binary encoded protos to worry about (which is typical) then you can rename fields.
shrubble 1 days ago [-]
"The main downside of this is the fact that object storage doesn't have transactions, so you can't make sure that both appending the event and updating the state of the world happen in one atomic unit. We write the state first and the event second, so a process dying in between leaves the state correct and the history missing an entry. Nothing detects that, because detecting it would mean something reads the log and compares it to state, and the entire point is that nothing does. So every entry in the log really happened. What you don't get is a guarantee that everything that happened is in it."
So, they built a thing that pretends to, but does not actually properly handle transactions?
I guess I should be glad they are not a fintech startup...
jmathai 1 days ago [-]
There is a simple version of this idea that really resonates with me.
For about a decade, I've been using flat files on disk or object storage for most of my side projects. There's even a python library that handles some of the plumbing for you [1].
If you don't have strong record-level concurrency needs then it's a lot nicer, easier, cheaper than a relational or document database. And if you do need that, you can design your data model around what defines a record.
> using a database engine we built on top of FoundationDB: a distributed key-value store
it would be interesting to see their experiences with using FoundationDB. this feels like a tech that is amazing if only it had more information and practical examples of how to use, leverage and manage it. the client is complicated and needs expertise to use correctly. would love it if there was more info about it all.
Absolutely no mention of Iceberg or Delta Lake, which have had many of the same properties for years. I wonder if they are aware of them, or otherwise why they invented something new.
I generally don't read low-quality AI slop though, so its possible I missed an explanation that didn't contain those key words.
1 days ago [-]
grey-area 1 days ago [-]
The moment a cache becomes load-bearing you have a database again, except it's in RAM, nobody backed it up, and its failure mode is silence.
Article flagged as AI slop.
quietraster 1 days ago [-]
building uniqueness and transactions on object storage is a bold trade. where did the operational complexity end up hurting most?
ecshafer 1 days ago [-]
Why? This just seems like a bad idea.
devin 1 days ago [-]
> Where it actually breaks
I had a sensible chuckle when I got to this part. This kind of article "____ is all you need" is like another case of Betteridge's Law of Headlines. The answer is "that's not true" every time.
jpsaccount 3 days ago [-]
Author here. Ampbase is an OpAMP control plane for agent fleets, and this is how it runs without a database: conditional writes for uniqueness and compare-and-swap, one bucket per customer so isolation isn't a WHERE clause somebody has to remember, ULID keys so history is a prefix list.
Two things the post doesn't cover and I'm happy to get into. What closing the cross-region lost update actually took: annotating the RPCs that depend on a compare-and-swap and replaying those to a single region, with a client-side guard. And the read amplification, which we have a plan for but waiting on a clear signal for when it’s needed.
Cross-posted with thanks to the Tigris folks; the original is at ampbase.io.
typesanitizer 1 days ago [-]
> [..] so isolation isn't a WHERE clause somebody has to remember, [..] Two things the post doesn't cover and I'm happy to get into. What closing the cross-region lost update actually took: [..]
Set of my Claude alarm bell, and lo-and-behold, Pangram judges this comment to be 100% AI-generated, albeit with limited confidence.
BenoitP 1 days ago [-]
There's even a load-bearing in there as well
themgt 1 days ago [-]
And the read amplification, which we have a plan for but waiting on a clear signal for when it’s needed.
This is Claude between the lines admitting none of it was needed. Postgres on a VM would be doing just fine right now.
8by3 1 days ago [-]
real people don't write "compare-and-swap" ... or "lo-and-behold" for that matter.
edit. Clearly the joke about "lo-and-behold" didn't land well with the bots...
ninjaoxygen 1 days ago [-]
People who grew up with assembly language or locking primitives are fairly likely to use compare-and-swap because it's literally the commonly understood term for it.
TIL, thanks for the context. What I was getting at still stands, LLM's seem to love a lot of words-with-hyphens which might be grammatically correct, but I've never seen a human actually type.
BenoitP 1 days ago [-]
Compare-and-swap is a hardware instruction, and beyond that a desirable pattern for dealing with concurrent writes.
For compare-and-swap I have even got a lock less queue that I implemented with it.
embedding-shape 1 days ago [-]
> and this is how it runs without a database
From the article:
> In practice, when you reach for a database engine you're actually reaching for four basic features: unique constraints, transactions, indices, and history tables. In order to use Tigris' global object storage as a database, we had to implement all of these primitives ourselves.
Not sure why the "without a database" is or isn't so important, why is it mentioned so often and why the article flip-flopping between "we don't have a DB" and "we're effectively building our own DB"?
To me it reads a bit like, how we built our office without desks: it turns out if you stack two chairs on top of each other, you can balance your laptop on the top and you’ll also have a shelf on the bottom for your things.
But that sounds kinda unstable and doesn't really provide the full benefits we expect from a...
Oh.
For those working with databases, it's easy to come up with a bunch of reasons. ORM is a necessary evil when working with RDBMS, as are things like schema migrations that can easily result in loss of data (I.e., dropping columns or tables).
What if all we need is dumping a big old JSON in a container?
This idea is very enticing. The scale of this whole NoSQL thing is pretty telling.
The blog presents a thought provoking question: what if we don't actually need a full-blown database, and instead we only care about things like unique constraints, transactions, indices, and history tables.
What if we only need a subset of those?
Hmm, neither of those been necessary for me. ORMs is a choice, you can choose not to and still have a proper design and architecture and not suffer from that choice. Requires you have programmers who know how to use SQL, which seems less and less common as the days go by though, so understandable most reach for an ORM.
Schema migrations that can "easily result" in data loss is even easier to avoid though. Don't drop the column/table in the same step you copy data to the new place, or do the migration via application code which gets cleaned up later, to catch most of the stuff before one bigger copy, and then the eventually DB cleanup.
Guess it depends on how careful you want to be, or how careful your agent lets you be, I guess. But it's definitely possible to avoid both of those "necessary evil", which I guess makes them just "evil".
See:
https://zodb.org
https://gemtalksystems.com/products/gs64/
This sound like: what if we dont actually need a table and instead we only care about four legs and a wooden plate.
Your time and attention is precious as a developer. I'm absolutely sure it's possible to implement uniqueness constraints, transactions, indices, and history yourself, but is that really the most valuable use of your time? There's probably not a need for you to have a unique solution, so you're quite literally just re-inventing something someone already had for not a lot of benefit.
Wouldn't your time be better spent actually solving the problems that whatever you're building is supposed to solve?
If you want to kill someone's dream of publishing a game, encourage them to build their own engine from scratch. I cannot think of a more malicious piece of advice given how effective it is, statistically speaking.
The exceptions to this are so incredibly rare. Virtually all of the in-house engine work at indie scale has been replaced by Godot in recent years. It used to be something like 10-11% of studios were in-house fully custom. Now it's probably closer to 1-2% fully custom and 8-10% on Godot. Having a reasonably stable OSS option has removed the last major argument that I am familiar with for simply using what already exists.
I was interested in gamedev as a kid. I always went with a custom engine. You're right, I never released/finished a large game.
But now I think I was just into game engine dev. Every time I finished my game engine and started working on game content I ingredient got bored and went to another project.
So I guess this is the reason, engine development is fun :).
Same vibe here imo. Use zerofs, juicefs. Use picomq, automq, s3stream. Use slatedb. Use lancedb, fusion, tonbo, iceberg, vertex. Use duckdb, data fusion, polars. These systems all natively speak to object storage. Use Qwikwit, rising wave / hummock, greptine mito, use warp stream, neon. Use celld. Over half of the projects here are very specifically about object storage.
Many deal with uniqueness, transactions, indicies. Which is extra impressive because its across machines. Managing data on a box creates a huge array of challenges and difficulties, backup and scaling and HA, etc etc. A huge reason to face object storage head on is that often you are headed towards that fray one way or another, and object storage makes an excellent easy to scale and manage disaggregation layer that keeps your problems from complecting together, is a clear, well known separation point. With an incredible ecosystem around it.
What if you just ran FoundationDB instead?
Would that let you do collaboration blog-posts with another VC-funded startup though?
In our current design we use FDB as our metadata store which is the sort of workload it is really good at.
Also, the database is the last thing you want to re-invent unless your business is explicitly building a db (even then it's best to re-use a db like all the postgresql forks that have existed)
good news, FoundationDB is already invented, and just slightly more tested than others on the market.
That might be a very specific assumption. What about serializability? Replication? Materialized views? Procedures? Locking? Access control?
It’s cool to experiment and try new approaches. Neat one here.
Could still end up moving to Postgres.
Did it take 10 seconds of prompting? Or hours of thought, trial error and revision? Especially when asking people to read for 20+ minutes...
If you only have binary encoded protos to worry about (which is typical) then you can rename fields.
So, they built a thing that pretends to, but does not actually properly handle transactions?
I guess I should be glad they are not a fintech startup...
For about a decade, I've been using flat files on disk or object storage for most of my side projects. There's even a python library that handles some of the plumbing for you [1].
If you don't have strong record-level concurrency needs then it's a lot nicer, easier, cheaper than a relational or document database. And if you do need that, you can design your data model around what defines a record.
[1] https://tinydb.readthedocs.io/en/latest/
it would be interesting to see their experiences with using FoundationDB. this feels like a tech that is amazing if only it had more information and practical examples of how to use, leverage and manage it. the client is complicated and needs expertise to use correctly. would love it if there was more info about it all.
I generally don't read low-quality AI slop though, so its possible I missed an explanation that didn't contain those key words.
Article flagged as AI slop.
I had a sensible chuckle when I got to this part. This kind of article "____ is all you need" is like another case of Betteridge's Law of Headlines. The answer is "that's not true" every time.
Two things the post doesn't cover and I'm happy to get into. What closing the cross-region lost update actually took: annotating the RPCs that depend on a compare-and-swap and replaying those to a single region, with a client-side guard. And the read amplification, which we have a plan for but waiting on a clear signal for when it’s needed.
Cross-posted with thanks to the Tigris folks; the original is at ampbase.io.
Set of my Claude alarm bell, and lo-and-behold, Pangram judges this comment to be 100% AI-generated, albeit with limited confidence.
This is Claude between the lines admitting none of it was needed. Postgres on a VM would be doing just fine right now.
edit. Clearly the joke about "lo-and-behold" didn't land well with the bots...
https://en.wikipedia.org/wiki/Compare-and-swap
CMPXCHG
https://www.felixcloutier.com/x86/cmpxchg
https://lwn.net/Articles/847973/
For compare-and-swap I have even got a lock less queue that I implemented with it.
From the article:
> In practice, when you reach for a database engine you're actually reaching for four basic features: unique constraints, transactions, indices, and history tables. In order to use Tigris' global object storage as a database, we had to implement all of these primitives ourselves.
Not sure why the "without a database" is or isn't so important, why is it mentioned so often and why the article flip-flopping between "we don't have a DB" and "we're effectively building our own DB"?