Depending on the culture you are living in, this can be perceived as either dismissive or respectful.
I was curious enough to browse through the comments of @lexicality to conclude that he is European, so there is a chance that you were considered capable of receiving a direct feedback.
That is, by giving direct feedback, @lexicality showed you respect.
Do you find it easy to find great, honest, and valid feedback on the internet?
How much of the supply of great, honest, and valid feedback would you sacrifice in order to make the supply that remains use "less dismissive"[0] language like "a bit silly"?
[0] Less dismissive in quotes because I do not find that language dismissive in the slightest
Let's chalk it up to cultural difference! Some commenters think "silly" is fine (Ireland, UK, US - although I'm from the US and I disagree, but let's grant it). Some think it's not.
Let's assume that "silly" was meant kindly and not dismissively!
Had never heard of SSE before. Followed the link and found this warning:
> Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100).
This is the first time I’ve heard of per-domain connection limits. Seems… not great? Doesn’t this turn into a client side DoS? User opens 6+1 tabs and now the browser has exhausted all HTTP connections to your domain serving SSE connections?
I’ve used long polling before, I don’t understand how I’ve never observed the connection limit of 6…
This limit was(/is) actually a common problem with HTTP/1.1, and was one of the motivations for HTTP/2. A common workaround is "domain sharding"[1] where you would split your site up between several domains, so that you could get more concurrent connections. Which is important for improving page load time, but doesn't really help with having SSE for multiple tabs (unless you do something crazy like using a randomly generated domain for each SSE connection).
> I don’t understand how I’ve never observed the connection limit of 6…
Practically speaking this limitation has been obsolete for 5 years or so. With HTTP/1.1 there used to be ways to achieve more parallel downloads through domain sharding [1] on the server side and browser tweaks [2] on the client side.
Perhaps we should define a new HTTP header for servers to say: "It's ok for clients to initiate N connections with me" so that we can write web apps without this restriction.
Hi! I am the author of http-feeds.org. Thank you for your feedback.
For this spec I aimed to keep it as simple as possible. And plain polling-based JSON Endpoints are the most simple and robust endpoints IMHO.
If you need, you could implement an SSE representation on the server endpoint by prober content negotiation.
The main reason, why I dropped SSE it the lack of proper back pressure, i.e. what happens when a consumes slower than the server produces messages.
Plus, it is quite hard to debug SSE connections, e. g. no support by Postman and other dev tools. And long lasting HTTP connections are still a problem in todays infrastructure. E. g. there is currently no support for SSE endpoints in Digital Ocean App Platform, and I am not sure about them in Google Cloud Run.
I'm not entirely sure what you mean by this. SSEs are just normal GET requests with a custom header and some formal logic around retries. I've even implemented them manually with PHP using the `retry` command to mean I don't need to have the connection open for longer than a normal pageload.
> The main reason, why I dropped SSE it the lack of proper back pressure, i.e. what happens when a consumes slower than the server produces messages.
Could you point me to where the spec handles this please? As far as I can tell it has the same problem of the server needing to buffer events until the client next connects
Thank you for writing this great spec up for others to use!
I think totally right that back-pressure and plain GETs are an important use-case to support, and am really happy to see a beautiful spec written up to articulate concretely how to support them.
It is also great to be able to switch amongst these methods of subscription, for instance, if your server can keep a persistent connection open, it's nice to be able to get realtime updates over a single channel, but to still be able to fall back to polling or long-polling if you can't. And if you switch between a polling and a subscription, it's nice if you don't have to change the entire protocol — but can just change the subscription method.
Maybe you'd be interested in incorporating your experience, use-cases, and design decisions into that effort? We have been talking about starting at this November's IETF. [1]
For instance, you can do polling over the Braid protocol [2] with a sequence of GETs, where you specify the version you are coming from in the Parents: header:
GET /foo
Parents: "1"
GET /foo
Parents: "2"
GET /foo
Parents: "3"
Each response would include the next version.
And you can get back-pressure over Braid by disconnecting a subscription when your client gets overwhelmed, and then reconnecting again later on with a new Parents: header:
GET /foo
Subscribe: true
Parents: "3"
..25 of updates flow back to the client..
**client disconnects**
Now the client can reconnect whenever it's ready for new data:
GET /foo
Subscribe: true
Parents: "28"
Or if it wants to re-fetch an old version, it can simply ask for it via the version ID it got:
GET /foo
Version: "14"
And if the source for these updates is a git repository, we could use a SHA hash for the version instead of an integer:
GET /foo
Version: "eac8eb8cb2f21c5e79c305c738aa8a8171391b36"
Parents: "8f8bfc8ea356d929135d5c3f8cb891031d1539bd"
There's a magical universality to the basic concepts at play here!
I had previously used SSE in a project and had no idea it existed or that it was widely supported by most browsers. It's a beautiful and straightforward technology, but it doesn't seem to be as widely used as it should be, and more tooling is required. Also more tooling is required.
Does SSE work reliably in mobile browsers nowadays? A few years back when I tried, it was working OK in Chrome for Android but mobile browser support didn't seem complete.
Having a standard for what events look like is a decent idea as people generally just make some random thing up without thinking too hard and having a standard is generally helpful - but making your own polling system etc when there's already a battle tested one available seems like wasted effort and a bit counterproductive
Very cool! This looks very similar to what we're doing with the braid spec[1], though I really like how clear and concise your examples are! This is a great little website.
Some differences between our approaches:
- I think its a good idea to support arbitrary patch formats via a content-type style field, just like we have different formats for images. This lets you use the same protocol for things like collaborative editors.
- For some data sets (like CRDTs), you want each change to be able to refer to multiple "parents".
- Your protocol is very JSON-y and not very HTTP-y. It looks like you're basically using JSON to express HTTP. Why not just use HTTP? One big downside of the JSON approach is that it makes it awkward to transmit binary patches. (Eg, 'patching' an image)
Feel free to reach out if you're up for a chat! Looks like we're working on the same problem.
This made me all warm and fuzzy inside that nerds can still help other nerds even when we are on different teams. Hope y'all are able to share some ideas.
It'd be great to get a fairly standard way of doing this. :)
Having worked in this problem space a bit recently, I find this part a bit too optimistic:
> The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.
The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer.
Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though.
Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences.
The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup)
A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.
Logical/vector clocks etc can be a real pain with ephemeral clients like web browsers, it can be hard to work out when it's safe to trim the now dormant nodes.
Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think. It definitely still comes up and there are things like CRDTs to help you address these cases (which do generally rely on either logical clocks or hybrid logical clocks). However most cases of event streaming to the browser you have each write round-tripping through the DB anyway and you use a feed like this to push a CDC stream or similar down into the browser to get "instant" feedback of a change that occurred after initial load.
JSON inside server sent events gives you a browser native client, no need for streaming extensions to your JSON lib which is kinda rare, and various intermediates know not to buffer the content type already. Sorry to be the what about X guy, maybe there's something I missed?
Yeah batched arrays isn't the way to go when when you want to mimic what you would get from a traditional message broker. This really should be a spec that builds on SSE but specifies the encoding (CloudEvents) and parameters to negotiate the serialisation (JSON/Protobuf/Avro) and lastEventId, then it would be much more useful.
I could see this being used as part of a distributed system for shopping. for example, say you have 20 stores, each with its own storage area for items, but you also have 2 large warehouses. Your online site might need to know which stores have stock now, or which warehouses have stock. The site might allow you to do "Click and collect" do it would need to know, in somewhat real time, which stores actually have the stock. each store would have its own endpoint with a data feed that the central server can get data from. Same with online orders for delivery. It needs to know what warehouses have the stock, and if none do, how to get the store to ship it to a customer.
Likewise, the stores might need to know what is in the warehouses, or even across town; someone walks in store to order something, but its not in that store. But they know, in real time, that its in the warehouse for delivery next day or that the store across the city has it.
What is done now is a more centralised approach; all sites would have a connection to a single DB in head office that stores everything. This makes things more distributed and, in theory, removes a single point of failure.
I should clarify, I am not working on any of this, this is just how I think it would work... if anyone wants to step in and tell me if I'm wrong, right, or just plan stupid, please shout.
I like this idea, however I think the event ID being encoded in the response body places constraints on what each element looks like. Perhaps it would make more sense to encode the last ID/feed position in a response header and have the client submit that in a subsequent request header? That would decouple the feed position from any one element or the response structure itself.
This seems similar, but not as fleshed out as RPDE feeds, which has already been accepted by the W3 [1], and provides many answers to lots of the 'eventually up to date' questions.
The idea is nice and needed. However maybe the spec is a bit elaborate for me to adopt it immediately. I've been rolling my own for some time at some clients, for our kafkaesque/event sourcing patterns.
However what I used there was simple http stream/json stream like this:
- No start of [] but JSON newline entries a new line is an new entry
- Using Anything as an id (we've been using redis XSTREAMS as lightweight kafka concepts, just 64bit integers)
- have an type as an event, and versioning is just done by upgrading the type, ugly, but easy.
- We'er considering using SSE at this moment
Compaction is not something that I would do in the protocol I think I would just expose another version of it on a different url I think or put it in a different spec.
How do you recover when there is e.g. some connection error and a client misses some events? Can the client ask to replay the events? Then, how far back can they go?
It's an interesting idea, but I wonder how well the promise will hold up that this is less complex than using a message broker.
Yes, the network topology and protocol are certainly less complex, but there are now additional strong requirements how an endpoint has to store and manage existing events. (See the bits about strict time ordering, compaction, aggregation, etc).
A lot of this is effectively what a message broker is doing in a "traditional" system to guarantee consistency. Those tasks aren't gone, they are just pushed to the endpoints now.
A few issues with message brokers, esp. in the system-to-system integration:
- Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ...
- Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, ... Who is on call when something goes wrong?
- Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...
> Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ...
That's a valid point. I think it's a pity we don't have an equivalent standard for asynchronous messaging with the same support as HTTP. However, there are lots of options for presenting an asynchronous public API that would use your message broker behind the scenes, without fully exposing it: Websockets, SSE, web hooks, etc...
> Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user,
True, but don't you have that anyway? How is this different from requesting a new database, service route, service definition, etc?
> Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...
How simple or complex this is depends on the concrete broker at hand. There are some protocols, e.g. STOMP that are simple enough that you could write your own client. And as I wrote in the parent: HTTP feeds are a technology as well. You'll have to think about troubleshooting and resiliency there as well.
My thoughts exactly. While this is more human readable on the wire, a message broker delivering the feed would provide many different other features that might be useful, such as transactions, load-balancing, guaranteed delivery and per-endpoint state to simplify the individual application instances.
For those not aware of what message brokers are, there are many to choose from such as: Mosquito, RabbitMQ, ActiveMQ, Solace... If delivery over HTTP is a requirement, many of these brokers support delivery over websockets or (in the case of Solace) also support long polling.
With more attention going to long polling again, I wonder if it would be useful to introduce some kind of HTTP signaling (header or 1xx status) to indicate on the protocol level that long polling is going on. This might be useful information for intermediaries - e.g. proxies, firewalls, browser network tab, etc.
My first thought is if the client does not specify a start ID it could be sent billions of records depending on the elapsed time and frequency of events. What would be the best way to avoid overloading a client?
How about "catching up" on events, if a client had an outage, can it indicate the last event ID it had and then get a natural replay of the older events?
Doh... thanks for pointing it out. I asked because it says elsewhere that persistence of objects is a non-goal, I guess it meant storing them clientside or something.
Indeed, these kinds of abbreviated notifications are a great way to implement realtime updates. The code is simpler and there's less chance of screwing something up.
I think serving data from multiple Kafka partitions gets unnecessarily hard if your "continue from this point" token is tied to a singular event ID. For that, it'd be better to have the "cursor token" be an arbitrary blob of data you repeat back to the server. Then the server can e.g. encode a list of partition->offset values into it.
I always implement my own standards. I am faster that way instead of learning what others created. Obviously this is only true for less complex subjects.
One obvious downside is that polling at a fixed interval is going to be less efficient than having an event "pushed" to your client as soon as it's available. If your events are low-volume, then there will be periods where your polling requests return no new data, but you still have to make those requests anyway in order to determine that. And conversely, if your events are high-volume, then your polling interval represents an arbitrary amount of delay that you're introducing into the system. That might not be big deal for some applications, but it's worth mentioning as a potential downside in situations where you want the behavior to be as near-instantaneous as possible.
Yeah there's no problems with a long-lived TCP connection unless one end of the connection tries to drop the connection after a certain amount of time (which certain firewalls do.)
Correct. No fixed sized pages, but dynamic batches based on the lastEventId.
This is much easier to implement, both in server and client side, and it greatly removed the amount of data transferred. With fixed pages you would return content of the latest page for every poll request until it is "full".
Feels like a bit silly not to have made this compatible with SSE since all the mechanisms for acessing that are built into browsers these days.
https://developer.mozilla.org/en-US/docs/Web/API/Server-sent... https://html.spec.whatwg.org/multipage/server-sent-events.ht...
> "Feels like a bit silly not to have made this compatible with SSE ..."
The feedback is great, and valid. I wish it could have been expressed less dismissively though.
Depending on the culture you are living in, this can be perceived as either dismissive or respectful.
I was curious enough to browse through the comments of @lexicality to conclude that he is European, so there is a chance that you were considered capable of receiving a direct feedback.
That is, by giving direct feedback, @lexicality showed you respect.
22 replies →
Do you find it easy to find great, honest, and valid feedback on the internet?
How much of the supply of great, honest, and valid feedback would you sacrifice in order to make the supply that remains use "less dismissive"[0] language like "a bit silly"?
[0] Less dismissive in quotes because I do not find that language dismissive in the slightest
Let's chalk it up to cultural difference! Some commenters think "silly" is fine (Ireland, UK, US - although I'm from the US and I disagree, but let's grant it). Some think it's not.
Let's assume that "silly" was meant kindly and not dismissively!
This reminds me of CometD - https://github.com/cometd/cometd. Is it same in concept?
Had never heard of SSE before. Followed the link and found this warning:
> Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100).
This is the first time I’ve heard of per-domain connection limits. Seems… not great? Doesn’t this turn into a client side DoS? User opens 6+1 tabs and now the browser has exhausted all HTTP connections to your domain serving SSE connections?
I’ve used long polling before, I don’t understand how I’ve never observed the connection limit of 6…
This limit was(/is) actually a common problem with HTTP/1.1, and was one of the motivations for HTTP/2. A common workaround is "domain sharding"[1] where you would split your site up between several domains, so that you could get more concurrent connections. Which is important for improving page load time, but doesn't really help with having SSE for multiple tabs (unless you do something crazy like using a randomly generated domain for each SSE connection).
[1]: https://developer.mozilla.org/en-US/docs/Glossary/Domain_sha...
> I don’t understand how I’ve never observed the connection limit of 6…
Practically speaking this limitation has been obsolete for 5 years or so. With HTTP/1.1 there used to be ways to achieve more parallel downloads through domain sharding [1] on the server side and browser tweaks [2] on the client side.
[1]: https://developer.mozilla.org/en-US/docs/Glossary/Domain_sha...
[2]: http://kb.mozillazine.org/Network.http.max-persistent-connec...
Perhaps we should define a new HTTP header for servers to say: "It's ok for clients to initiate N connections with me" so that we can write web apps without this restriction.
4 replies →
Hi! I am the author of http-feeds.org. Thank you for your feedback.
For this spec I aimed to keep it as simple as possible. And plain polling-based JSON Endpoints are the most simple and robust endpoints IMHO.
If you need, you could implement an SSE representation on the server endpoint by prober content negotiation.
The main reason, why I dropped SSE it the lack of proper back pressure, i.e. what happens when a consumes slower than the server produces messages. Plus, it is quite hard to debug SSE connections, e. g. no support by Postman and other dev tools. And long lasting HTTP connections are still a problem in todays infrastructure. E. g. there is currently no support for SSE endpoints in Digital Ocean App Platform, and I am not sure about them in Google Cloud Run.
Overall, plain GET endpoints felt much simpler.
I'm not entirely sure what you mean by this. SSEs are just normal GET requests with a custom header and some formal logic around retries. I've even implemented them manually with PHP using the `retry` command to mean I don't need to have the connection open for longer than a normal pageload.
> The main reason, why I dropped SSE it the lack of proper back pressure, i.e. what happens when a consumes slower than the server produces messages.
Could you point me to where the spec handles this please? As far as I can tell it has the same problem of the server needing to buffer events until the client next connects
Thank you for writing this great spec up for others to use!
I think totally right that back-pressure and plain GETs are an important use-case to support, and am really happy to see a beautiful spec written up to articulate concretely how to support them.
It is also great to be able to switch amongst these methods of subscription, for instance, if your server can keep a persistent connection open, it's nice to be able to get realtime updates over a single channel, but to still be able to fall back to polling or long-polling if you can't. And if you switch between a polling and a subscription, it's nice if you don't have to change the entire protocol — but can just change the subscription method.
There's a growing effort to integrate all our various subscription-over-http protocols into an IETF HTTP standard, that we can use for all these use-cases: https://lists.w3.org/Archives/Public/ietf-http-wg/2022JanMar...
Maybe you'd be interested in incorporating your experience, use-cases, and design decisions into that effort? We have been talking about starting at this November's IETF. [1]
For instance, you can do polling over the Braid protocol [2] with a sequence of GETs, where you specify the version you are coming from in the Parents: header:
Each response would include the next version.
And you can get back-pressure over Braid by disconnecting a subscription when your client gets overwhelmed, and then reconnecting again later on with a new Parents: header:
Now the client can reconnect whenever it's ready for new data:
Or if it wants to re-fetch an old version, it can simply ask for it via the version ID it got:
And if the source for these updates is a git repository, we could use a SHA hash for the version instead of an integer:
There's a magical universality to the basic concepts at play here!
[1] https://www.ietf.org/how/meetings/115/
[2] https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
Yeah, we're also pushing for SSE for this kind of things.
I had previously used SSE in a project and had no idea it existed or that it was widely supported by most browsers. It's a beautiful and straightforward technology, but it doesn't seem to be as widely used as it should be, and more tooling is required. Also more tooling is required.
4 replies →
Does SSE work reliably in mobile browsers nowadays? A few years back when I tried, it was working OK in Chrome for Android but mobile browser support didn't seem complete.
1 reply →
Having a standard for what events look like is a decent idea as people generally just make some random thing up without thinking too hard and having a standard is generally helpful - but making your own polling system etc when there's already a battle tested one available seems like wasted effort and a bit counterproductive
Very cool! This looks very similar to what we're doing with the braid spec[1], though I really like how clear and concise your examples are! This is a great little website.
Some differences between our approaches:
- I think its a good idea to support arbitrary patch formats via a content-type style field, just like we have different formats for images. This lets you use the same protocol for things like collaborative editors.
- For some data sets (like CRDTs), you want each change to be able to refer to multiple "parents".
- Your protocol is very JSON-y and not very HTTP-y. It looks like you're basically using JSON to express HTTP. Why not just use HTTP? One big downside of the JSON approach is that it makes it awkward to transmit binary patches. (Eg, 'patching' an image)
Feel free to reach out if you're up for a chat! Looks like we're working on the same problem.
[1] https://github.com/braid-org/braid-spec/blob/master/draft-to...
This made me all warm and fuzzy inside that nerds can still help other nerds even when we are on different teams. Hope y'all are able to share some ideas.
It'd be great to get a fairly standard way of doing this. :)
Having worked in this problem space a bit recently, I find this part a bit too optimistic:
> The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.
The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer.
Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though.
Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences.
The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup)
A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.
Perhaps the time problem can be handled with an eventually consistent Lamport clock system
Logical/vector clocks etc can be a real pain with ephemeral clients like web browsers, it can be hard to work out when it's safe to trim the now dormant nodes.
Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think. It definitely still comes up and there are things like CRDTs to help you address these cases (which do generally rely on either logical clocks or hybrid logical clocks). However most cases of event streaming to the browser you have each write round-tripping through the DB anyway and you use a feed like this to push a CDC stream or similar down into the browser to get "instant" feedback of a change that occurred after initial load.
3 replies →
JSON inside server sent events gives you a browser native client, no need for streaming extensions to your JSON lib which is kinda rare, and various intermediates know not to buffer the content type already. Sorry to be the what about X guy, maybe there's something I missed?
Yeah this feels very much like someone retyped the Server Sent Events spec and bolted an arbitrary spec on for each message...
Yeah batched arrays isn't the way to go when when you want to mimic what you would get from a traditional message broker. This really should be a spec that builds on SSE but specifies the encoding (CloudEvents) and parameters to negotiate the serialisation (JSON/Protobuf/Avro) and lastEventId, then it would be much more useful.
Can someone ELI5 the significance of this?
* What are some scenarios where you would need a feed?
* What's being done now to solve the problem and how is this different?
Many thanks
This basically puts a REST/HTTP GET frontend on your messaging backend (Kafka, RabbitMQ, MQTT, etc).
So whatever events you want to expose publicly can be done over plain HTTP.
Having said that, IMO using SSE or websockets would be a better fit than raw HTTP.
I could see this being used as part of a distributed system for shopping. for example, say you have 20 stores, each with its own storage area for items, but you also have 2 large warehouses. Your online site might need to know which stores have stock now, or which warehouses have stock. The site might allow you to do "Click and collect" do it would need to know, in somewhat real time, which stores actually have the stock. each store would have its own endpoint with a data feed that the central server can get data from. Same with online orders for delivery. It needs to know what warehouses have the stock, and if none do, how to get the store to ship it to a customer.
Likewise, the stores might need to know what is in the warehouses, or even across town; someone walks in store to order something, but its not in that store. But they know, in real time, that its in the warehouse for delivery next day or that the store across the city has it.
What is done now is a more centralised approach; all sites would have a connection to a single DB in head office that stores everything. This makes things more distributed and, in theory, removes a single point of failure.
I should clarify, I am not working on any of this, this is just how I think it would work... if anyone wants to step in and tell me if I'm wrong, right, or just plan stupid, please shout.
Great answer and exactly what I was looking for! Thank you
I like this idea, however I think the event ID being encoded in the response body places constraints on what each element looks like. Perhaps it would make more sense to encode the last ID/feed position in a response header and have the client submit that in a subsequent request header? That would decouple the feed position from any one element or the response structure itself.
This seems similar, but not as fleshed out as RPDE feeds, which has already been accepted by the W3 [1], and provides many answers to lots of the 'eventually up to date' questions.
[1] https://www.w3.org/2017/08/realtime-paged-data-exchange/
The idea is nice and needed. However maybe the spec is a bit elaborate for me to adopt it immediately. I've been rolling my own for some time at some clients, for our kafkaesque/event sourcing patterns.
However what I used there was simple http stream/json stream like this:
- No start of [] but JSON newline entries a new line is an new entry
- Using Anything as an id (we've been using redis XSTREAMS as lightweight kafka concepts, just 64bit integers)
- have an type as an event, and versioning is just done by upgrading the type, ugly, but easy.
- We'er considering using SSE at this moment
Compaction is not something that I would do in the protocol I think I would just expose another version of it on a different url I think or put it in a different spec.
How do you recover when there is e.g. some connection error and a client misses some events? Can the client ask to replay the events? Then, how far back can they go?
It's an interesting idea, but I wonder how well the promise will hold up that this is less complex than using a message broker.
Yes, the network topology and protocol are certainly less complex, but there are now additional strong requirements how an endpoint has to store and manage existing events. (See the bits about strict time ordering, compaction, aggregation, etc).
A lot of this is effectively what a message broker is doing in a "traditional" system to guarantee consistency. Those tasks aren't gone, they are just pushed to the endpoints now.
A few issues with message brokers, esp. in the system-to-system integration:
- Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ... - Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, ... Who is on call when something goes wrong? - Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...
> Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ...
That's a valid point. I think it's a pity we don't have an equivalent standard for asynchronous messaging with the same support as HTTP. However, there are lots of options for presenting an asynchronous public API that would use your message broker behind the scenes, without fully exposing it: Websockets, SSE, web hooks, etc...
> Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user,
True, but don't you have that anyway? How is this different from requesting a new database, service route, service definition, etc?
> Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...
How simple or complex this is depends on the concrete broker at hand. There are some protocols, e.g. STOMP that are simple enough that you could write your own client. And as I wrote in the parent: HTTP feeds are a technology as well. You'll have to think about troubleshooting and resiliency there as well.
My thoughts exactly. While this is more human readable on the wire, a message broker delivering the feed would provide many different other features that might be useful, such as transactions, load-balancing, guaranteed delivery and per-endpoint state to simplify the individual application instances.
For those not aware of what message brokers are, there are many to choose from such as: Mosquito, RabbitMQ, ActiveMQ, Solace... If delivery over HTTP is a requirement, many of these brokers support delivery over websockets or (in the case of Solace) also support long polling.
This seems like a really attractive way of publishing events to a third party. Platform independent and easy to understand.
The platform independence is implicit in the "HTTP" part of the name.
With more attention going to long polling again, I wonder if it would be useful to introduce some kind of HTTP signaling (header or 1xx status) to indicate on the protocol level that long polling is going on. This might be useful information for intermediaries - e.g. proxies, firewalls, browser network tab, etc.
My first thought is if the client does not specify a start ID it could be sent billions of records depending on the elapsed time and frequency of events. What would be the best way to avoid overloading a client?
I think the server will give a truncated response.
They need to clearly document what is the behaviour for this case.
This could form the foundation of a distributed social media network.
How about "catching up" on events, if a client had an outage, can it indicate the last event ID it had and then get a natural replay of the older events?
That's the second example on the page.
Doh... thanks for pointing it out. I asked because it says elsewhere that persistence of objects is a non-goal, I guess it meant storing them clientside or something.
I wrote a blog post advocating for something like this recently, with an additional push to notify the client when it should poll: https://blog.separateconcerns.com/2022-03-05-push-to-poll.ht...
I wasn't aware of this but it fits the use case perfectly, I will update the post.
Indeed, these kinds of abbreviated notifications are a great way to implement realtime updates. The code is simpler and there's less chance of screwing something up.
I've also seen this described as hints: https://blog.andyet.com/2014/09/23/send-hints-not-data/
I think serving data from multiple Kafka partitions gets unnecessarily hard if your "continue from this point" token is tied to a singular event ID. For that, it'd be better to have the "cursor token" be an arbitrary blob of data you repeat back to the server. Then the server can e.g. encode a list of partition->offset values into it.
I always implement my own standards. I am faster that way instead of learning what others created. Obviously this is only true for less complex subjects.
See also: technical debt
Not if you keep using your standards across different projects
This looks great. Has anyone any insights or reference about the limits of this approach compared to more traditional message queues?
One obvious downside is that polling at a fixed interval is going to be less efficient than having an event "pushed" to your client as soon as it's available. If your events are low-volume, then there will be periods where your polling requests return no new data, but you still have to make those requests anyway in order to determine that. And conversely, if your events are high-volume, then your polling interval represents an arbitrary amount of delay that you're introducing into the system. That might not be big deal for some applications, but it's worth mentioning as a potential downside in situations where you want the behavior to be as near-instantaneous as possible.
Way back I recall (ab)using chunked encoding to do something similar. Now there's an underexploited element of HTTP...
Is it sensible to keep the TCP connection alive for long periods of time, reading in events as ljson? I imagine that this is no issue for QUIC
Yeah there's no problems with a long-lived TCP connection unless one end of the connection tries to drop the connection after a certain amount of time (which certain firewalls do.)
Sounds good. Thanks for pointing out the issue with some firewalls
No pagination?
You probably mean "maximum batch size", because pagination is handled by the lastEventId and the fact that a stream is always consumed in order.
Correct. No fixed sized pages, but dynamic batches based on the lastEventId.
This is much easier to implement, both in server and client side, and it greatly removed the amount of data transferred. With fixed pages you would return content of the latest page for every poll request until it is "full".