If you are building a protocol, it serves to think in _layers_.
How will this be represented in a database?
How will that be retrieved?
How will this be represented in a message?
How will this be represented in a UI?
etc.
You don't have to _specify_ all of these, but you should have a story for them and that story needs to take into account scalability and capability, and it needs to have considered the tradeoffs involved.
Example.
1/
Awhile back I built an API and a junior dev asked why not build a single API call that would allow the UI to load everything it needed in one go. The idea being to reduce the QPS of the service
We talked it over. The reason was because I was approaching the problem in _layers_
What parts of the UI are tolerant to waiting? Which are not?
How big of a payload can an individual API call take for our customers?
How much complexity is there in the message?
What does it look like in the db?
2/
It also considers questions like: what happens if the UI changes in the future? What kind of change surfaces exist? Can we remove or add elements easily?
etc
In that project, loading data could be potentially arduous, requiring scanning tens if not hundreds of GB for some parts of the UI, but requiring trivial amounts of data for other parts.
This is part of why some protocols go so far as to specify the format of the data that you are storing, or provide structures for efficient storage
3/
Does this mean every protocol needs to have broken down these questions?
Kind of. Yes. Even if it is never published or formalized, you need to think about "how will this protocol represent data, how will data get loaded into it, how will it be stored?"
If you aren't doing that, you are missing a key element in how protocols need to be considered.
4/
@hrefna One thing I wish protocols did was have a standard for doing even basi8c filters over collections of data. E.g. I think AP pull-based sync could be made much faster if we had a standard for filtering by time from the client-side. In general I feel a lot of protocols that have append only logs in them are sorely lacking indexing at the sync layer.
@hrefna TIL, ty for the link. On my end we've been experimenting with something similar to MongoDB queries but represented via querystring parameters. Nothing formal or public yet though.
@mauve This is part of why Google standardized on CEL: to provide a safe and consistent way to do filtering in the protocol request.