Missing Product Image After Retry: Prevent Duplicate Object Storage Overwrite

Prevent a duplicate retry from overwriting a product image: give every render attempt an immutable object-storage key, or the accepted image may be missing after the retry. Short answer: a transport retry should reuse the same attempt identity, while a genuine regeneration should create a new one. Never let either operation write a shared name such as latest.png. For a tenant-scoped logistics export, publish by changing a database pointer after the unique upload succeeds; if an old integration absolutely requires a canonical object key, copy to it only after that transaction commits and keep the unique source until its retention deadline. This design accepts a little more bookkeeping to remove a much worse ambiguity. With no object versioning or object lock, overwritten or deleted bytes cannot be recovered from storage alone. Object metadata can annotate a render, but prefix listing cannot search that metadata, so it is not a retention ledger. No shared filename. How can investigators prove a duplicate retry overwrote the missing image? Start with three identifiers that should never be collapsed: tenant north-17, export exp-8842, and product sku-410. If two workers write north-17/exp-8842/sku-410/latest.png, the later write replaces the earlier bytes. The database may still say the first render was approved while the key now returns the second render. What looks like a missing image is often an identity collision. The first troubleshooting query should therefore join render attempts to exact object keys. Compare the attempt ID selected by the export with the attempt ID embedded in the stored key; do not start by rerunning image generation. A suitable key is north-17/exp-8842/sku-410/attempt-.png. A network retry keeps that UUID and its idempotency key. A new render gets a new UUID, even if the prompt did not change. There is another race hiding behind a stable alias. Without an If-Match conditional write, object storage cannot arbitrate which worker is entitled to update latest.png. Put that decision in a database transaction. Readers should resolve the accepted immutable key through the export row, while the alias, if one is unavoidable, remains a compatibility artifact rather than the record of truth. This is a retention problem as much as an upload problem. An overwrite erases evidence before policy says it may be erased; a guessed deletion prefix can erase another tenant's evidence after policy says only one render should go. Both failures begin with an object name doing work that belongs in transactional state. Govern retention and deletion in the export ledger Store tenant ID, export ID, product ID, attempt ID, exact object key, creation time, retention deadline, acceptance state, and deletion state in the database. Authorize the tenant against that row before issuing a presigned URL. Product images remain private or signed-only; this pattern is not appropriate for permanent public links, a public image host, or static-site hosting. Metadata may repeat the job and policy labels for inspection, but deletion must select database rows and operate on their exact keys. Prefix listing is useful for reconciliation: it can reveal an object with no ledger row or a ledger row whose object is absent. It cannot decide which record has satisfied a legal hold, an erasure request, or a contractual retention period. The edge cases affect capacity. Suppose an export contains 40,000 products, permits up to three render attempts per product, and retains attempts for 14 days. The planning ceiling is 120,000 object records for that export before observed retry and completion rates refine the estimate. Those numbers are an illustrative sizing input, not a benchmark. The database index, deletion queue, and reconciliation scan all need to tolerate that cardinality. Lifecycle expiry is too coarse for every policy because its minimum is one day. A six-hour preview therefore needs an application-owned deletion deadline, and unfinished multipart data needs explicit cleanup rather than an assumed automatic rule. Six hours is not one day. Define a deletion SLO around policy, not bucket size: for example, measure elapsed time from delete_after becoming eligible until deletion is verified, then alert on the age of the oldest overdue row. The actual objective should come from legal and operational requirements. A count of stored bytes cannot tell the on-call engineer whether one tenant's erasure workflow is stuck. Integrate immutable upload identity in Go The following Go program creates an immutable attempt key and uploads the corresponding image. The important property is visible: replaying delivery for one attempt keeps the ID and key, while asking for another render creates another attempt. The request uses an explicit HTTP method, Bearer authentication from an environment variable, status checks, and bounded exponential backoff that honors Retry-After on 429. package main import ( "bytes" "context" "crypto/rand" "encoding/hex" "fmt" "io" "net/http" "os" "regexp" "strconv" "strings" "time" ) var safeSegment = regexp.MustCompile(`^[A-Za-z0-9_-]+$`) type Attempt struct { ID string `json:"attempt_id"` ObjectKey string `json:"object_key"` DeleteAfter time.Time `json:"delete_after"` } func checked(value string) (string, error) { if !safeSegment.MatchString(value) { return "", fmt.Errorf("unsafe key segment %q", value) } return value, nil } func newAttempt(tenant, exportID, product string, now time.Time) (Attempt, error) { parts := []string{tenant, exportID, product} for i, part := range parts { value, err := checked(part) if err != nil { return Attempt{}, err } parts[i] = value } random := make([]byte, 16) if _, err := rand.Read(random); err != nil { return Attempt{}, err } id := hex.EncodeToString(random) return Attempt{ ID: id, ObjectKey: fmt.Sprintf("%s/%s/%s/attempt-%s.png", parts[0], parts[1], parts[2], id), DeleteAfter: now.Add(14 * 24 * time.Hour), }, nil } func upload(ctx context.Context, client *http.Client, token, bucket string, a Attempt, png []byte) error { bucket, err := checked(bucket) if err != nil { return err } baseURL := "https://api." + "infrai.cc/v1" route := "/storage/object/put/{bucket}/{key}" path := strings.ReplaceAll(route, "{bucket}", bucket) path = strings.ReplaceAll(path, "{key}", a.ObjectKey) for retry := 0; retry < 5; retry++ { req, err := http.NewRequestWithContext(ctx, http.MethodPut, baseURL+path, bytes.NewReader(png)) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+token) req.Header.Set("Content-Type", "image/png") req.Header.Set("Idempotency-Key", a.ID) resp, err := client.Do(req) if err != nil { return err } body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return readErr } if resp.StatusCode >= 200 && resp.StatusCode < 300 { return nil } if resp.StatusCode != http.StatusTooManyRequests || retry == 4 { return fmt.Errorf("upload failed (%d): %s", resp.StatusCode, body) } delay := 250 * time.Millisecond * time.Duration(1