Hacker News Reader

Comment by aaaaaaaaaab | Hacker News Reader

“Once the originally desired resource is created, there are two alternative ways to deal with the temporary task resource:

API client must issue DELETE request, so that server purges it. Until then, server responds with 303 See Other status. Once deleted, 404 Not Found will be returned for subsequent GET /queue/12345 requests. or, garbage collection can be a server’s job to do: once task is complete server can safely remove it and respond with the 410 Gone on subsequent GET /queue/12345 requests.”

How do you distinguish between an id referring to a finished task that no longer exists (410 Gone), and an id that never referred to any task (404 Not Found)? Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale...

View on HN

https://tools.ietf.org/html/rfc7231#section-6.5.9

> It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.

I'd say "gone" is a temporary state that exists shortly after task completion. A task shows as "gone" for, say, 15min, and then is deleted, producing a 404 in subsequent requests.

On further thought, it seems to me that we necessarily need to keep completed tasks around until asked by a client to delete them. If a task produces an exception, we need to notate that for when the client next asks about the task's status. If the task ran successfully to completion, we need to notate that for when the client next asks. If the client needs to know all this, then it's the client who needs to ask for deletions as well - we shouldn't assume it's OK to remove records about tasks (unless there's a published policy that completed/error data will be purged after a certain period, e.g. 30, 60, 90 days...)

I’d say after a period of tolerance (410) you switch to 404. If you care about historicals, move the record to a dedicated DW solution and use that to report.

> Other than the obvious solution of keeping every task in the DB with a “finished” flag, which doesn’t scale...

Why doesn't that scale?

Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.

If your IDs are completely unique, you could add them to a Bloom filter [0] when cleaning up the resource. Then you could use that to determine if an incoming ID has possibly been used before or definitely not.

[0] https://en.wikipedia.org/wiki/Bloom_filter

Or you could just have your IDs be monotonically increasing and look at the minimum. Any task prior to that number is "410 Gone" by definition.

Bloom filters are probabilistic, meaning they don't always produce the right result. From the article you posted:

> False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed (though this can be addressed with a "counting" filter); the more elements that are added to the set, the larger the probability of false positives.

So I don't think they're the right choice for this problem, if you actually depend on the result "not here" vs "never was here" being right.

You could keep it very tight. IME it’s the index overhead that kills (1000 times as much data makes inserts 10x slower), and ever since partial indexes became a thing that’s a bit less of a problem.

Also deleting is far from free of side effects.

> Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.

The only info that needs to be stored is the ID of the finished task. If, say, the ID is expressed as a GUID then we're talking about 4 ints or 2 longs worth of data.

Even so, these records are supposed to be garbage collected.

So? Storage is cheap. Even supposing you were going to store every single one as a 16 byte UUID, you can store 2^32 such ids in 2^32*2^4=2^36 bytes=~64GB. Is that a large database in your mind?

Sure, but what's wrong with that?

Reply on news.ycombinator.com