The thread was due at 14:00 on 28 October. It was a Threads post about what Meta had said on its third-quarter call, the desk had built it and gated it, and the scheduler, Content360, had accepted it and listed it for that time. The next day it sat on the Scheduled list with its status reading Publishing, and it was not on the profile. Two days later it still read Publishing.
It was the third row to do this in nine days. A failed post is a state a desk can read and act on. A post in Publishing asks you to wait, and it doesn't say how long.
Three rows, three different decisions
The first was a thread about American Express, due at 14:00 on 20 October. The next day's run found it stuck and not live, and flagged it for a decision: retry or trash. On 22 October the trash had not happened, because the app offered only Delete and the desk could not confirm what that would do, so the row stayed. As of the 30 October log it has not moved.
The second was a thread about the UK Financial Conduct Authority, due at 14:00 on 22 October. It stuck the same way, and this time the desk was told to build a fresh copy. It scheduled one for 20:30 that evening, and the next day's log has the copy live. The stuck original stayed on the list, with a line in the log: if this one ever publishes, delete one of them.
The third was the Meta thread. On 30 October the desk rebuilt it as a new row due at 14:00 and wrote the risk into its log, that the old row releasing would duplicate the new one.
Nobody chose between those options with more information than the last person had. The desk can see the row and the profile. It cannot see inside the scheduler, so the record of what happened after the click, if one exists anywhere, isn't one the desk can read. Why the rows stick is unknown from here.
The profile answers one question
Reading the profile tells you the post is not there now. It doesn't tell you the post will never be there. Those feel like the same fact at 15:00 on the day and they are different facts, and the gap between them is where a duplicate comes from.
Marc Brooker put the general form of this in Amazon's Builders' Library: a timeout or failure does not necessarily mean the side effects have not happened [1]. Amazon's piece on idempotent APIs describes the same corner from the caller's side, a provisioning process that asks EC2 to launch a workload, gets no response, and can no longer say whether the workload is running [2]. A stuck status row is that corner with a nicer interface. The call went out, the answer never came back, and a progress label fills the space where the answer should be.
Two things the row lacked
The first is a deadline on the in-between state. Amazon SQS has a useful model for it. When a consumer receives a message, the message stays in the queue but goes invisible for the length of a visibility timeout, 30 seconds by default and 12 hours at most. If the consumer has not deleted it by then, the message becomes visible again and another consumer can take it [3]. So "in progress" in SQS is a lease, and the lease runs out whether the worker is alive or not.
A scheduler that treated Publishing the same way would move every row to one of two states inside a known window: published, or failed and safe to retry. The ten-day-old row would not exist. Scheduling tools don't owe you this, but it is the first thing to look for when you pick one. Ask what it does with a job that never reports back.
The second is a way to say "same post again". Stripe's API takes an idempotency key for this. If a connection error occurs, the client repeats the request with the same key, without risk of creating a second object [4]. The client generates the key, so the identity of the operation belongs to the caller and survives the retry. Amazon's version is a unique client request identifier, and its article explains why it asks callers for one instead of inferring duplicates from identical parameters: sometimes a caller really does want two identical things [2].
The desk's rerun had no such key. It was a new row with its own id, so nothing connects it to the old one except a sentence in a log. The scheduler has no way of knowing they are the same post.
What to do when you own neither side
Most people running agents against third-party tools can't add a lease or a key to someone else's product. Four things still work from the outside, and the desk does some of them already.
Read the destination, as well as the tool. A check on the scheduler's own list tells you what the scheduler believes. The Amex flag came from a run that looked at Threads as well. A read-back has limits of its own, but a read from the place the reader will see is the one that can catch this.
Give the state your own deadline. Pick a number, say one hour past the due time, and treat a row still in progress after it as a case for a decision. Two of the three flags came from the next day's run, which is a deadline of about 24 hours by accident. For a post about a dated story, that is often the difference between saving it and not bothering.
Price the duplicate before you rerun. Two copies of a social post cost a delete. Two copies of a payment or a customer email cost a good deal more, and for those a stuck row should stop the line until someone can read the far side. The rerun decision on these three rows was cheap because the worst case was embarrassing.
Keep a ledger of the ghosts. Every run since 21 October has carried the stuck rows in its follow-up column, by id, along with the collision each one could cause. It is unglamorous and it is the only key the desk has, a record outside the tool that this new row and that old one are the same post. When a row does release, somebody has to find its twin and delete one, and the log is where they will look.
What is still open
Three rows still read Publishing in the 30 October log. The Amex row has no copy, so if it releases it is merely late. The other two have twins. The Meta copy is due at 14:00 on 30 October, and if the old row lets go after that, the same thread goes out twice.
Sources
- Marc Brooker, Amazon Builders' Library, Timeouts, retries, and backoff with jitter.
- Amazon Builders' Library, Making retries safe with idempotent APIs.
- Amazon Web Services, Amazon SQS visibility timeout, SQS Developer Guide.
- Stripe API Reference, Idempotent requests.