Sync dog breed images to your own S3 bucket

Sync dog breed images to S3 with a short Python script: for each wanted breed and view, check whether the key already exists in your bucket, download the full-size PNG once if it does not, upload it under breed/variant/view.png, and print your usage at the end.

Why keep the files yourself

Dog Image API sells full-size images by a monthly count of distinct images. One image is one view of one set at 2048 x 2048 pixels. Full-size files are meant to be downloaded once and stored on your side, and you should not link your pages to the API for them.

That shapes the design:

• Your app reads from your bucket, so it keeps working if you later change plans.
• The script asks the API only for keys the bucket does not hold yet.
• The key never leaves the machine that runs the script. It is not in a browser or an app.

The images are AI-generated illustrations of one dog per set, in nine views, on a transparent background. If your app labels where an image came from, say so.

Prepare

Install the libraries and set three environment variables. The AWS credentials come from the usual boto3 chain, such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, a profile, or a role.

pip install boto3 requests
export DOG_IMAGE_API_KEY="paste-your-key-here"
export S3_BUCKET="my-dog-images"
export AWS_DEFAULT_REGION="us-east-1"

The bucket should be private to start. Put a CDN or signed URLs in front of it when you serve the files.

The script

Save it as sync_dog_images.py.

import os
import sys
import time

import boto3
import requests
from botocore.exceptions import ClientError

BASE = "https://dogimage.dev"
BUCKET = os.environ["S3_BUCKET"]
API_KEY = os.environ["DOG_IMAGE_API_KEY"]
PREFIX = "dogs"

# What you want to keep. The breed's main set is used for every breed.
BREEDS = ["pug", "beagle", "samoyed"]
VIEWS = ["front", "left", "head"]

s3 = boto3.client("s3")


class QuotaUsedUp(Exception):
    pass


def error_text(resp):
    try:
        err = resp.json()["error"]
        return f"{resp.status_code} {err['code']}: {err['message']}"
    except (ValueError, KeyError):
        return f"{resp.status_code} {resp.reason}"


def already_stored(key):
    try:
        s3.head_object(Bucket=BUCKET, Key=key)
        return True
    except ClientError as err:
        if err.response["Error"]["Code"] in ("404", "NoSuchKey", "NotFound"):
            return False
        raise


def fetch_full(breed, variant, view, attempts=4):
    url = f"{BASE}/api/v1/images/{breed}/{variant}/{view}"
    for attempt in range(1, attempts + 1):
        resp = requests.get(
            url,
            params={"size": "full"},
            headers={"Authorization": f"Bearer {API_KEY}"},
            timeout=60,
        )
        if resp.status_code == 200:
            return resp.content, resp.headers.get("X-Quota-Remaining")
        if resp.status_code == 429:
            raise QuotaUsedUp(error_text(resp))
        if resp.status_code in (502, 503) and attempt < attempts:
            time.sleep(2**attempt)
            continue
        raise RuntimeError(error_text(resp))


def print_usage():
    resp = requests.get(
        f"{BASE}/api/v1/usage",
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=30,
    )
    if resp.status_code != 200:
        print("Usage check failed:", error_text(resp))
        return
    u = resp.json()
    limit = u["limit"] if u["limit"] is not None else "no limit"
    print(f"Usage for {u['month']} on plan {u['plan']}: {u['used']} used, limit {limit}.")


def main():
    catalog = requests.get(f"{BASE}/api/v1/breeds", timeout=30).json()
    by_slug = {b["slug"]: b for b in catalog["breeds"]}
    uploaded = skipped = 0
    remaining = None
    failed = False

    try:
        for breed in BREEDS:
            if breed not in by_slug:
                sys.exit(f"Unknown breed slug: {breed}")
            variant = by_slug[breed]["default_variant"]
            for view in VIEWS:
                key = f"{PREFIX}/{breed}/{variant}/{view}.png"
                if already_stored(key):
                    skipped += 1
                    continue
                data, remaining = fetch_full(breed, variant, view)
                s3.put_object(
                    Bucket=BUCKET,
                    Key=key,
                    Body=data,
                    ContentType="image/png",
                    CacheControl="public, max-age=31536000, immutable",
                )
                uploaded += 1
                print(f"uploaded s3://{BUCKET}/{key}")
    except QuotaUsedUp as err:
        print(f"Stopping early: {err}")
    except RuntimeError as err:
        print(f"Stopped on an error: {err}")
        failed = True

    print(f"{uploaded} uploaded, {skipped} already in the bucket.")
    if remaining is not None:
        print(f"X-Quota-Remaining on the last download: {remaining}")
    print_usage()
    sys.exit(1 if failed else 0)


if __name__ == "__main__":
    main()

Run it with python sync_dog_images.py.

How it decides what to fetch

The key layout

Each file lands at dogs/<breed>/<variant>/<view>.png, for example dogs/pug/fawn/front.png. The script reads default_variant from the breed list and uses that real color slug, so the name in your bucket does not depend on what "default" means later. The API accepts the color slug in the same position as default.

Skipping what you hold

head_object is a cheap check that returns a not-found error when the key is missing. Only that error means "download it". Any other error, such as a permission problem, is raised so you see it instead of re-downloading everything.

Quota, 429 and retries

Every full-size reply carries X-Quota-Limit, X-Quota-Used and X-Quota-Remaining. The script keeps the last remaining value and prints it. On the All images plan the header reads unlimited, which is why it is printed as text.

A 429 quota_used_up ends the loop without a traceback. Files already uploaded stay, and the final report still runs. A 502 or 503 is retried after 2, 4 and 8 seconds. The errors and quotas reference covers the rest.

The usage report

GET /api/v1/usage needs the key and returns month, plan, limit, used and remaining. On the All images plan, limit is null, and the script prints "no limit". This call is the number to watch before you add more breeds. Check pricing for the plan sizes.

Serving from the bucket

Your own copy of a full-size file needs no credit in your product. The license still forbids reselling or republishing the images as a collection, a dataset or an image library, so do not make the bucket public as a download source. For caching ideas around previews and stored files, see cache and self-host dog images.