Skip to content

Sources

A source turns a target you type into download tasks. downctl picks the source automatically from the target's URI scheme; --source NAME forces one.

Scheme Source Target form
https:// http:// url https://host/path/file
hf:// (or bare org/name) huggingface hf://org/name[@revision]
gh:// github gh://owner/repo[@tag]
s3:// s3 s3://bucket/key or s3://bucket/prefix/
sftp:// sftp sftp://[user@]host[:port]/path
file:// file file:///abs/path

Common filters --include PAT (repeatable) and --exclude PAT apply to the listing sources (huggingface, github, s3 prefixes, sftp directories, file directories).

downctl get https://example.com/big.iso
downctl get https://example.com/a.bin https://example.com/b.bin -j 2
  • Filename is the final path segment (URL-decoded, basename only).
  • Server Range support enables multi-connection + resume; if a server ignores Range it falls back to a single stream automatically.

huggingface — repositories

downctl get hf://meta-llama/Llama-3-8B
downctl get hf://meta-llama/Llama-3-8B --include '*.safetensors' --exclude '*consolidated*'
downctl get hf://datasets/squad@main
downctl get org/name                 # bare repo id also works
downctl get hf://org/repo@v1.0 -d models
  • Lists the repository via the Hub API (models and datasets).
  • Uses each file's LFS sha256 for post-download verification.
  • --revision REV or @REV pins a revision (default main).
  • Options ([plugins.huggingface]):
  • token — access token for private repos / higher limits (or env HF_TOKEN / HUGGING_FACE_HUB_TOKEN). Sent only to the HF host.
  • mirror — base host instead of https://huggingface.co, e.g. https://hf-mirror.com.

github — release assets

downctl get gh://cli/cli
downctl get gh://cli/cli --include '*.tar.gz'
downctl get gh://owner/repo@v1.2.3
  • Lists assets of the latest release (or a @tag).
  • Uses the asset sha256 digest when the API provides one.
  • Options ([plugins.github]):
  • token — API token (or env GITHUB_TOKEN / GH_TOKEN). With a token, assets are fetched through the API asset URL (Accept: application/octet-stream) so the credential stays on the API host and never leaks to the download CDN.
  • api — API base URL (GitHub Enterprise).

s3 — S3-compatible storage

downctl get s3://my-bucket/path/to/file.bin
downctl get s3://my-bucket/datasets/ --include '*.parquet'
  • Objects are fetched through SigV4 presigned URLs (stdlib only), so the normal HTTP transport handles ranges, multiple connections and resume.
  • A prefix is listed with ListObjectsV2 (paginated).
  • Credentials come from [plugins.s3] or the standard AWS env vars; the presigned query is redacted from logs and the failed log.
  • Options: region, access_key, secret_key, session_token, endpoint, path_style (default true), expires (1–604800 s).

Example with MinIO:

[plugins.s3]
endpoint = "http://127.0.0.1:9000"
region = "us-east-1"
access_key = "minioadmin"
secret_key = "minioadmin"
path_style = true

sftp — over SSH (optional dependency)

pip install 'downctl[sftp]'
downctl get sftp://user@host:22/home/user/data.bin
downctl get sftp://user@host/data/ --include '*.csv'
  • A trailing / (or a directory) is listed; a file is downloaded directly.
  • The SFTP transport honours byte ranges (seek), so ranges/multi-connection/ resume all work.
  • Host keys are verified against ~/.ssh/known_hosts when present; known_hosts = "" (or none/false) disables verification.
  • Credentials may be in the URL (user:pass@) or in [plugins.sftp] (user, password, key, key_passphrase, port, known_hosts). Passwords in URLs are stripped from tasks and logs.

file — local files

downctl get file:///data/models/        # copy a directory recursively
downctl get file:///data/a.bin -d /backup
  • Directories are walked recursively; --include/--exclude filter paths.
  • Uses a local file transport with range support, so the same pipeline (bitmap, resume) applies — handy for copying into another directory.