M: Can I tell the gallery-dl utility to add webm and mp4 files
G: Yes, you can configure
gallery-dl to download specific file types, including
webm and mp4, by adjusting its configuration
file (config.json). Here’s how:
gallery-dl:Locate or create the config.json file. The file is
typically found in:
~/.config/gallery-dl/config.json%APPDATA%\gallery-dl\config.jsonAdd or modify the format section to include the
desired file extensions. For example:
{
"extractor": {
"base-directory": "~/Downloads",
"archive": "archive.sqlite3",
"filename": "{category}/{author}/{id}.{extension}",
"postprocessors": [
{
"name": "metadata"
}
]
},
"format": {
"file": {
"extensions": ["webm", "mp4"]
}
}
}Restart or re-run gallery-dl to apply the
changes.
If you want to filter downloads by file type on a per-run basis, use
the --filter option:
gallery-dl "URL" --filter "extension in ('webm', 'mp4')"This ensures only webm and mp4 files are
downloaded.
M: Can I add a delay between each file downloaded of 2 seconds?
G: Yes, you can configure
gallery-dl to add a delay between each downloaded file by
using the sleep option in the configuration file or
specifying it in the command line.
config.jsonAdd the sleep-request option under the
extractor section of your config.json
file:
{
"extractor": {
"sleep-request": 2.0
}
}This setting introduces a 2-second delay between each request made by
gallery-dl.
If you want to set the delay on a per-run basis, use the
--sleep-request option:
gallery-dl --sleep-request 2 "URL"M: Sleep delay isn’t working
G: If --sleep-request
or "sleep-request" in config isn't working as expected,
here are likely reasons and correct alternatives:
sleep-request delays between HTTP
requests, not between files.sleep instead of sleep-requestsleep adds a delay between each
download, which is what you want.
config.json:{
"extractor": {
"sleep": 2.0
}
}This tells gallery-dl to wait 2 seconds between each
item.
gallery-dl --sleep 2 "URL"--verbose to verify sleepRun with --verbose to confirm it's pausing:
gallery-dl --sleep 2 --verbose "URL"You’ll see log lines indicating sleep time between items.
Let me know if it still skips the delay after that.
M: i swear i asked you about this, but let's start over because i lost all my data... i'm trying to run gallery-dl, configuring it to replace uncommon/problematic characters with underscores, delay a couple seconds to prevent server overloads, and automatically number things including leading zeros. like, let's say i wanted to use a file named 'list.txt' containing URLs like "https://boards.4chan.org/w/thread/2207908/ranma-12-papes" to get a decent wallpaper collection; check online, then walk me through what i need to do.
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: reasoning_recap; content_type=reasoning_recap]
G: TL;DR — Yes. Put your URLs in
list.txt, create a config.json that (1)
replaces ugly/path-breaking characters with _, (2) sleeps
~2s between files, and (3) zero-pads numbers in filenames. Then run:
gallery-dl -i list.txtDetails below, including Fedora install and a ready-to-paste config.
# Fedora repo
sudo dnf install gallery-dl
# …or latest via pipx (keeps it isolated)
python3 -m pip install --user pipx && pipx ensurepath
pipx install gallery-dlWhere config files live (any of these works):
~/.config/gallery-dl/config.json,
~/.gallery-dl.conf, or /etc/gallery-dl.conf.
citeturn3search8
Create list.txt (one URL per line), e.g.:
https://boards.4chan.org/w/thread/2207908/ranma-12-papes
Then you can run all of them with -i list.txt (reads
from file/stdin). citeturn0search3turn0search8
config.jsonThis gives you:
{
"base-directory": "~/Downloads/gallery-dl",
"extractor": {
"path-restrict": {
" ": "_",
"/": "_",
"\\": "_",
"|": "_",
":": "_",
"*": "_",
"?": "_",
"\"": "_",
"<": "_",
">": "_"
},
"path-remove": "\u0000-\u001f\u007f",
"sleep": 2.0,
"sleep-request": 0.0,
"sleep-extractor": 0.0,
"#": "Default filename: site/post-or-thread/NNN.ext with zero padding",
"filename": "{num:0>3}.{extension}",
"directory": ["{category}", "{title|id}"]
}
}Why these options work:
path-restrict/path-replace/path-remove
control sanitizing; using a mapping lets us force spaces →
_ and flatten other problematic characters to
_. Defaults and special forms are documented in the
manpage. citeturn2view0sleep is the between-file delay you
actually feel; sleep-request is between individual HTTP
requests during extraction (often not what you want for throttling
downloads). citeturn2view0filename accepts Python-style formatting.
{num:0>3} yields 001, 002, …
123. The docs show formatting on numeric fields (e.g.,
{page:>03} in examples); zero-padding is the same idea,
with the 0> fill/alignment. citeturn3search10# Basic: read URLs from list.txt and go
gallery-dl -i list.txt
# Show what keys you can use in filenames for a given URL (super helpful)
gallery-dl -K "https://boards.4chan.org/w/thread/2207908/ranma-12-papes"-K/--list-keywords prints the variables available for
that extractor (so you can refine your naming). citeturn2view0
directory to include board/thread id (e.g.,
["4chan", "{board}", "{thread}"]) once you see the exact
keys via -K. citeturn4search8sleep is
per file; use --verbose to see logs confirming sleeps. And
if a site serves many files per single HTTP request,
sleep-request won’t help with between-file pacing.
citeturn2view0gallery-dl --sleep 2 --path-restrict "windows" --path-replace "_" -i list.txt"windows" set replaces characters
invalid on Windows; config variant above additionally changes spaces to
_.) citeturn2view0If you want, I can tune the filename to include board name, thread
ID, post time, and keep the zero-padded {num} so your
wallpapers sort perfectly.
[Attachment reference unresolved: file-names]
[Attachment reference unresolved: file-naming]
[Attachment reference unresolved: file-album]
M: additions: i want any image tags to be saved to the image (or if not there, then to a separate file); the first item in the file name should be (with leading zeros if necessary) the post number, for absolute comparison; and tell me if there's a way to avoid redundancy - like, if i was doing an image search at danbooru, and wanted to first get everything with the tags "xenosaga -short_hair" and then everything with the tag "xenosaga", but not download the stuff already acquired, is there a built in method of doing this?
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: reasoning_recap; content_type=reasoning_recap]
G: TL;DR — Yes. You can 1)
embed a 2s delay between files, 2) sanitize
filenames to underscores, 3) start filenames with a
zero-padded post ID, 4) save tags (JSON
sidecar and/or a plain “tags.txt”), and 5) avoid
re-downloading when you run multiple Danbooru queries by using
the download archive. Run with -i list.txt
and the config below.
config.jsonPut this at ~/.config/gallery-dl/config.json:
{
"base-directory": "~/Downloads/gallery-dl",
"extractor": {
"sleep": 2.0,
"path-restrict": "ascii",
"path-replace": "_",
"archive": "~/Downloads/gallery-dl/archive.sqlite3",
"filename": "{num|id:0>9}.{extension}"
},
"postprocessors": [
{ "name": "metadata", "mode": "json", "event": "after" },
{ "name": "metadata", "mode": "tags", "event": "after" }
]
}What this does:
sleep: 2.0
inserts ~2 seconds after each downloaded file. (This is different from
sleep-request, which is between HTTP requests.)
citeturn0search15path-restrict: "ascii" strips non-ASCII;
path-replace: "_" turns anything disallowed by the
restriction into _. You can also use a custom map if you
want to target specific characters.
citeturn0search4turn0search9archive records downloaded IDs so repeats are skipped
automatically—even if you run broader Danbooru searches later. Same
effect if you use --download-archive FILE.
citeturn0search20{num|id:0>9} tries num first (e.g., 4chan
post number), else falls back to id (e.g., Danbooru post
id), formatting as 9 digits with leading zeros (000000123). gallery-dl
supports Python’s format mini-language for this.
citeturn0search2metadata postprocessor
writes sidecars.
mode: "json" → a .json next to each file
with all fields (including tags where provided).mode: "tags" → a plain text file (newline-separated
tags).Tip: For Danbooru you’ll get rich
tagsin metadata; 4chan won’t have image tags, but you’ll still get per-post info in the JSON. citeturn0search15
list.txtYour list.txt might contain:
https://boards.4chan.org/w/thread/2207908/ranma-12-papes
Run:
gallery-dl -i list.txtIf you prefer CLI flags instead of config for some parts:
gallery-dl -i list.txt \
--sleep 2 \
--download-archive ~/Downloads/gallery-dl/archive.sqlite3The archive ensures that when you later run something broader (e.g.,
xenosaga -short_hair first, then xenosaga),
already-seen IDs are skipped, so no duplicates are
downloaded. citeturn0search20
num."filename": "{num:0>9}.{extension}"id:
"filename": "{id:0>9}.{extension}"You can confirm the available keys per URL with:
gallery-dl -K "https://boards.4chan.org/w/thread/2207908/ranma-12-papes"
gallery-dl -K "https://danbooru.donmai.us/posts?tags=xenosaga"(-K/--list-keywords shows the exact fields you can use.)
citeturn0search16
If you want the tags inside the image metadata (instead of
sidecars), use the exec postprocessor to call
exiftool after each download. Example sketch:
{
"postprocessors": [
{ "name": "metadata", "mode": "json", "event": "after" },
{
"name": "exec",
"event": "after",
"command": [
"exiftool",
"-overwrite_original",
"-charset", "filename=utf8",
"-XMP-dc:Subject<=${metadata[tags]|[]|join(',',_)}",
"{filepath}"
]
}
]
}(The pattern above assumes tags are available; tune the expression to
your preference. See community examples of using exec +
ExifTool.) citeturn0search8turn0search13
archive in config or
--download-archive FILE on the CLI). It records each post’s
ID, so a later broader tag search won’t fetch
duplicates again. citeturn0search20If you prefer a custom mapping instead of blanket
ASCII stripping, use a dict mapping in path-restrict (e.g.,
turn spaces and any specific symbols into _). Example:
"extractor": {
"path-restrict": {
" ": "_",
"/": "_",
"\\": "_",
"\"": "_",
"<": "_",
">": "_",
"|": "_",
":": "_",
"*": "_",
"?": "_"
},
"path-remove": "\u0000-\u001f\u007f",
"path-replace": "_"
}See maintainer guidance on path-restrict strategies.
citeturn0search4
If you want, tell me whether you’re prioritizing 4chan or Danbooru
first and I’ll tailor the exact
filename/directory to include board, thread,
or tag slices while keeping the post ID first for
strict sorting.
[Attachment reference unresolved: file-names]
M: one more thing - what if there's a particular string i want to append to every url? like, with danbooru, if i wanted to append "+-pencil" to prevent any pictures with pencils to every search, could i do that simply?
G: Good question — there isn’t a
single “append this to every URL” switch in gallery-dl, but
there are two easy ways to get the effect you want for Danbooru (or
other query-string–based extractors):
Simplest: edit or generate list.txt so every query
already has the suffix you want, e.g.:
https://danbooru.donmai.us/posts?tags=xenosaga+-pencil
https://danbooru.donmai.us/posts?tags=xenosaga+-short_hair+-pencil
If you don’t want to edit by hand, you can wrap it with a shell one-liner:
# Take list.txt, append "+-pencil" to every line, pipe into gallery-dl
sed 's/$/+-pencil/' list.txt | gallery-dl -i -This way gallery-dl never sees the unmodified URL.
--exec as a
wrapperAnother trick is to preprocess on the fly: you can run gallery-dl in a loop where you transform the URL before handing it over. For example in Bash:
while read -r url; do
gallery-dl "${url}+-pencil"
done < list.txtThat appends +-pencil every time without altering
list.txt.
gallery-dl’s config doesn’t have a built-in “always
append this query string,” but if you want to bake it in permanently for
Danbooru, you can wrap gallery-dl with a little shell alias:
alias danbooru-dl='gallery-dl "https://danbooru.donmai.us/posts?tags="and then supply only the base tags. Or create a tiny wrapper script
that adds your +-pencil automatically before calling
gallery-dl.
For your specific case (exclude pencil in every search),
the sed trick is the least effort — it works for any
existing list.txt, and you don’t have to modify the config.
If you’re going to do it often, a wrapper script is better so you never
forget.
👉 Do you want me to sketch out a Bash wrapper (like
gdl-danbooru) that you can drop in ~/bin and
then always type gdl-danbooru xenosaga to get the right URL
with +-pencil appended? That way you never touch the raw
list.
M: what if i want to use a local custom json for a particular directory
G: gallery-dl
can use a local, directory-specific custom
JSON config — you just need to invoke it correctly.
There is no automatic “per-directory config discovery”,
but you can easily achieve the effect.
Below are all workable methods, from cleanest to hackiest.
-c ./config.json inside that
directoryThis is officially supported.
If you have:
/home/user/wallpapers/
custom-config.json
list.txt
Then you run:
cd /home/user/wallpapers
gallery-dl -c custom-config.json -i list.txtgallery-dl will use that JSON only for this
run.
This is the cleanest, simplest, most reliable method.
Example: /home/user/wallpapers/gdl.sh
#!/bin/bash
gallery-dl -c "$PWD/custom-config.json" "$@"Make executable:
chmod +x gdl.shNow simply use:
./gdl.sh -i list.txtThis makes the directory “self-contained”.
If you always run gallery-dl inside the directory, create an alias in
~/.bashrc or ~/.zshrc:
alias gdlw='gallery-dl -c /home/user/wallpapers/custom-config.json'Then:
cd /home/user/wallpapers
gdlw -i list.txtIf you want gallery-dl to always load your custom JSON when run from that shell:
export GALLERY_DL_CONFIG="/home/user/wallpapers/custom-config.json"Then run:
gallery-dl -i list.txtNote: This affects the whole shell session, not just the directory.
You can fake “per-directory config auto-loading” with a tiny launcher script:
Create /usr/local/bin/gallery-dl-local:
#!/bin/bash
# If config.json exists in cwd, use it.
if [[ -f "./config.json" ]]; then
gallery-dl -c ./config.json "$@"
else
gallery-dl "$@"
fisudo chmod +x /usr/local/bin/gallery-dl-localUse it instead of gallery-dl:
gallery-dl-local -i list.txtNow any directory can have its own config.json,
and it will be auto-used.
gallery-dl does not automatically load
configs from the current directory.gallery-dl -c ./config.jsonIf you want, I can build you a ready-made directory-local config loader that also:
+-pencil, etc.)Just tell me.
M: i tried piecing together things again step by step, but likely missed a comma or something. can you please review and give me a functional version? check online if you need to - but all i need is syntax correction, no other modification.
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: reasoning_recap; content_type=reasoning_recap]
G: Here’s a syntactically valid version of your config, with only the minimum fixes:
"filename".extractor block and
"postprocessors"."#" key as-is (it’s a valid JSON key, not a
comment).path-restrict ("ascii")
entry, because JSON cannot have duplicate keys. I kept your
explicit mapping version instead so your custom replacements still work.
fileciteturn0file0{
"base-directory": "~/Downloads/gallery-dl",
"extractor": {
"path-restrict": {
" ": "_",
"/": "_",
"\\": "_",
"|": "_",
":": "_",
"*": "_",
"?": "_",
"\"": "_",
"<": "_",
">": "_"
},
"path-remove": "\u0000-\u001f\u007f",
"path-replace": "_",
"sleep": 2.0,
"sleep-request": 0.0,
"sleep-extractor": 0.0,
"archive": "~/Downloads/gallery-dl/archive.sqlite3",
"filename": "{num|id:0>9}.{extension}",
"#": "Default filename: site/post-or-thread/NNN.ext with zero padding",
"directory": ["{category}", "{title|id}"]
},
"postprocessors": [
{ "name": "metadata", "mode": "json", "event": "after" },
{ "name": "metadata", "mode": "tags", "event": "after" },
{
"name": "exec",
"event": "after",
"command": [
"exiftool",
"-overwrite_original",
"-charset",
"filename=utf8",
"-XMP-dc:Subject<=${metadata[tags]|[]|join(',',_)}",
"{filepath}"
]
}
]
}You can drop this in as config.json and
gallery-dl should parse it without complaints.
M: okay, currently it's trying to run, but every item translates the filename to 'None.[ext]', where [ext] is gif or whatever, and then skips everything but the first item since there's already a file with that name.
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
M: i think you may have started to think too deeply. take a deep breath and tell me what you think this json configuration file is supposed to do, based on the configuration i uploaded and what you remember from this silo; it was started a while ago, so you might not actually have everything i think you have to work with. remember, this is for gallery-dl and you have permission to go online if necessary.
G: Understood — stepping back,
clearing the slate, and treating this as “What is this config
intended to do?”
Not “what is wrong,” not “how to fix it,” but “what was the
purpose of this configuration based on what you uploaded and what we
discussed earlier?”
Here is the high-level interpretation, without overthinking:
Based on the JSON you uploaded and the earlier goals in this silo, this config is intended to achieve five major behaviors:
Your path-restrict block:
"path-restrict": {
" ": "_",
"/": "_",
"\\": "_",
"|": "_",
":": "_",
"*": "_",
"?": "_",
"\"": "_",
"<": "_",
">": "_"
}This means:
_.Purpose:
Make filenames and directories safe and consistent everywhere (Linux,
Windows, future archives, etc.).
"path-remove": "\u0000-\u001f\u007f"This strips ASCII control characters.
Purpose:
Avoid bizarre rare characters appearing in filenames (common with
scraped text).
"sleep": 2.0This is the between-files delay, what you explicitly wanted to prevent server hammering.
Purpose:
Slow down downloads ~2 seconds per file so you don’t accidentally hammer
services (4chan, Danbooru).
"filename": "{num|id:0>9}.{extension}"This means:
{num} first (4chan post number).num doesn’t exist, try {id} (Danbooru
post ID).jpg,
png, gif, webm, etc.).Purpose:
"archive": "~/Downloads/gallery-dl/archive.sqlite3"This is specifically for your earlier question:
If I download “xenosaga -short_hair” and then “xenosaga”, can I avoid overlap?
Purpose:
Record every downloaded file’s ID so future runs skip duplicates.
Your postprocessors block asks for three things:
{ "name": "metadata", "mode": "json", "event": "after" }Produces:
000001234.json
Containing everything gallery-dl knows (IDs, tags, timestamps,
etc.).
{ "name": "metadata", "mode": "tags", "event": "after" }Produces:
000001234.tags
A simple newline-separated tag list.
"name": "exec",
"command": [ "exiftool", ..., "{filepath}" ]This attempts to write tags directly into the image’s XMP metadata.
Purpose:
"directory": ["{category}", "{title|id}"]Meaning:
4chan).Purpose:
Organize downloads sanely.
“Download everything safely, delay politely, name everything by post-number, guarantee no duplicates across searches, sanitize filenames, save tags separately, embed metadata if possible, and organize output into human-readable folders.”
This matches exactly what you’ve been aiming for in this silo:
Just describing intention, but noting the mismatch:
Your filename rule:
"{num|id:0>9}.{extension}"assumes every extractor provides either num
or id.
If neither exists for a particular site or sub-item (thumbnails,
inline links, removed posts, etc.),
gallery-dl substitutes None.
So:
Your config is correct in purpose — the site simply didn’t
provide the fields your rule depended on.
(And since every file becomes None.ext, everything after
the first gets skipped due to name collision.)
If you want, now that your intentions are clarified cleanly, I can generate a corrected version that:
None.ext,Just tell me to proceed.
M: okay, keeping things isolated instead of trying to fix every possible variation, tell me how - if it is possible - to modify the current version of the config file to follow the current rule when the information is available - but to use default information if it is not; if this is not possible, tell me so. then, try and tell me why when i was trying to update gallery-dl with dnf (sudo dnf update gallery-dl), it didn't work - like, did i use bad syntax, or is the utility discontinued, or..?
G: Below is the clean, direct answer, keeping everything isolated, no overreach, and no modifying unrelated parts.
{num} and {id} are missing?Short answer:
Yes — but only if you supply the fallback yourself.
gallery-dl does not have a built-in
automatic “default variable” for filenames.
The syntax you used:
{num|id:0>9}
means:
numidBut if both are missing, gallery-dl inserts the
Python None object → "None".
To avoid this, you must explicitly give a fallback value:
"{num|id|filename:0>9}.{extension}"
OR even safer, if you want a guaranteed numeric sequence:
"{num|id|index:0>9}.{extension}"
Where:
filename = original filename without extension (if
provided by extractor)index = a guaranteed sequential counter assigned by
gallery-dl for each itemResult:
numidfilename or index,
never NoneYES, it is possible to modify your config so it ONLY uses your desired rule when available, and a safe default otherwise.
This is the simplest, cleanest, and officially supported fallback mechanism.
sudo dnf update gallery-dl not work?Because dnf update X is not a valid
command to update a single package.
Fedora / DNF syntax is not like apt.
sudo dnf upgradesudo dnf upgrade gallery-dldnf update gallery-dl fails:update exists but is
an alias for the legacy upgrade only when used
without specifying a single package name.sudo dnf update gallery-dl
DNF interprets it as:
Fedora’s default repo sometimes lags behind the latest gallery-dl release.
If the version in the repo is already “the latest according to Fedora,” then:
dnf upgrade gallery-dl
will simply respond:
Nothing to do.
This does NOT mean gallery-dl is discontinued — only that:
Use pipx, which cleanly isolates it:
pipx install gallery-dl
pipx upgrade gallery-dlThis avoids the lag between upstream releases and Fedora packages.
✔ Yes. Use:
"{num|id|index:0>9}.{extension}"
You’ll never see None.ext again.
✔ Because dnf update package isn’t correct
syntax,
and Fedora’s gallery-dl packages often lag behind
upstream.
If you want, I can generate a clean final config with the fallback included and nothing else changed.
M: (base) [david@new-host-1 ____gallery]$ pip upgrade gallery-dl ERROR: unknown command "upgrade" (base) [david@new-host-1 ____gallery]$ pip3 upgrade gallery-dl ERROR: unknown command "upgrade" (base) [david@new-host-1 ____gallery]$ pipx upgrade gallery-dl bash: pipx: command not found... Similar command is: 'pip' (base) [david@new-host-1 ____gallery]$ pip update gallery-dl ERROR: unknown command "update" (base) [david@new-host-1 ____gallery]$ sudo dnf upgrade gallery-dl Updating and loading repositories: Repositories loaded. Failed to resolve the transaction: No match for argument: gallery-dl You can try to add to command line: --skip-unavailable to skip unavailable packages (base) [david@new-host-1 ____gallery]$
G: You didn’t do anything wrong — this is just the exact intersection of three different systems (pip, pipx, and DNF) each behaving correctly but differently from what their names suggest.
Below is the straight explanation, no fluff, no glossing:
pip upgrade / pip update DOESN’T WORKBecause pip does not have upgrade/update
subcommands.
The only valid way to upgrade a pip-installed package is:
pip install --upgrade gallery-dlor:
pip3 install --upgrade gallery-dlThat’s it.
pip upgrade will always throw:
ERROR: unknown command "upgrade"
pipx DOESN’T WORKYou got this:
bash: pipx: command not found
That means pipx is not installed.
Install it like this:
sudo dnf install pipx
pipx ensurepathThen:
pipx install gallery-dl
pipx upgrade gallery-dlpipx is optional, but by far the cleanest way to run
command-line tools isolated from system packages.
dnf upgrade gallery-dl FAILEDYour error:
No match for argument: gallery-dl
This means Fedora’s repos on your system do not contain gallery-dl at all.
On Fedora, gallery-dl is usually packaged as:
python3-gallery-dl
Try:
sudo dnf install python3-gallery-dlTo upgrade it later:
sudo dnf upgrade python3-gallery-dlIf that still says “No match,” then your current Fedora version / repo set simply doesn’t include it.
Given the errors you saw, it is almost certain that:
(base)), which means pip installs go inside condaCheck:
which gallery-dlYou’ll probably see something like:
/home/david/miniconda3/bin/gallery-dl
or:
~/.local/bin/gallery-dl
Either way → DNF will never see it, because it was not installed through the Fedora package manager.
Since you’re clearly running in a conda (base) shell,
use pip inside that environment:
pip install --upgrade gallery-dlIf gallery-dl was installed with pip3 instead:
pip3 install --upgrade gallery-dlIf it was installed under --user:
pip install --upgrade --user gallery-dlAfter that:
gallery-dl --version| Tool | Command to upgrade gallery-dl | Why yours failed |
|---|---|---|
| pip | pip install --upgrade gallery-dl |
pip has no upgrade or update
subcommand |
| pipx | pipx upgrade gallery-dl |
pipx wasn’t installed |
| dnf | sudo dnf upgrade python3-gallery-dl |
your repo doesn’t provide gallery-dl package |
If you want, I can walk you through:
Just tell me.
M: does the json format have a comment token
G: Short
answer:
No. JSON does not support comments of
any kind.
There is no official comment token, not #,
not //, not /* */.
This is why:
# ...// .../* ... */will all cause strict JSON parsers to throw an error.
Certain libraries (like json5, or some config loaders
for VSCode, Node.js, etc.) allow:
// comment
/* comment */
But that is JSON5, not JSON.
gallery-dl uses strict JSON, not
JSON5.
(You already did this, and it is the correct method.)
Example:
{
"#": "This is a comment",
"extractor": { ... }
}or
{
"_comment": "Explanation goes here",
"extractor": { ... }
}Because JSON doesn’t forbid random keys, this is 100% valid.
This is the standard hack used in config files for programs that require pure JSON.
Not ideal, but safe.
| Format | Comments Allowed? | Tokens |
|---|---|---|
| JSON (what gallery-dl uses) | ❌ No | None |
| JSON5 | ✔ Yes | //, /* */ |
| YAML | ✔ Yes | # |
| TOML | ✔ Yes | # |
| INI | ✔ Yes | ;, # |
If you'd like, I can help you restructure your config so “pseudo-comments” are grouped cleanly, readable, and impossible to conflict with real keys.
M: it's working-ish - i just took out the filename stuff entirely to see how the rest was functioning - but there are two things. this has popped up some, not interrupting but i'd like to know if there's a fix: [formatter][error] Invalid format string '-XMP-dc:Subject<=${metadata[tags]|[]|join(',',_)}' (ValueError: Empty attribute in format string) [postprocessor][error] 'exec' initialization failed: ValueError: Empty attribute in format string
...and there's also a danbooru-type site that keeps doing this: AuthRequired: 'api-key' & 'user-id' needed to access the API ('Missing authentication.')
G: Here is the clear, focused explanation of both issues — no overthinking, just what’s causing them and what your options actually are.
exec postprocessor formatting errorYou’re seeing:
[formatter][error] Invalid format string '-XMP-dc:Subject<=${metadata[tags]|[]|join(',',_)}'
(ValueError: Empty attribute in format string)
[postprocessor][error] 'exec' initialization failed
Because gallery-dl’s formatter does NOT allow nested
expressions inside ${...} when used in an
exec command block.
This part:
${metadata[tags]|[]|join(',',_)}
is using advanced filters (join, fallback, array indexing).
That syntax is valid in filenames or metadata templates, but NOT
inside the exec-command formatter.
Inside exec commands, gallery-dl tries to parse your
string with Python’s str.format() rules → which
cannot parse:
${metadata[...]}join(',',_)|[] fallback filtersHence:
ValueError: Empty attribute in format string
ExifTool tag embedding cannot use gallery-dl templating in
this way.
Exif metadata embedding can work — but not
with that template syntax.
${tags} if the
site exposes it directly)Right now:
Your configuration is correct everywhere else; only the
exec=EXIF block is invalid.
You’re seeing something like:
AuthRequired: 'api-key' & 'user-id' needed to access the API ('Missing authentication.')
This is not an error in your config.
This is the site itself.
Some Danbooru-type sites (e.g., Gelbooru, Moebooru forks, Philomena forks, Sankaku, etc.) require:
to access:
Without those credentials, the extractor switches to:
Inside your gallery-dl config:
{
"extractor": {
"danbooru": {
"user-id": "YOUR_USER_ID",
"api-key": "YOUR_API_KEY"
}
}
}or for whichever site is giving the error:
"gelbooru": { ... }
"sankaku": { ... }
"moebooru": { ... }
"philomena": { ... }The exact field names differ per extractor.
If you tell me the URL of the site causing the message, I can give the exact block needed.
There is no fix — the site is intentionally blocking unauthenticated API usage.
But gallery-dl will still download the images if the site
allows non-API scraping.
Some Danbooru forks do not allow this; they fully require
login.
exec
command array.If you want:
Just tell me which direction you prefer.
M: kind of thought that last bit. ... is there any gallery-dl mode which causes it to function more directly like a browser to parse things and decide what to download that way, rather than using the api? and if not, is there a workaround - either an alternate but similar utility, or some form of simple wrapper? (don't give the code if you get that far - just confirm/deny the possibility.)
G: Here is the direct, unambiguous answer to each part:
Yes — but only for some sites.
gallery-dl has two kinds of extractors:
For Danbooru-type sites: absolutely not.
The maintainers intentionally do not provide an HTML
fallback for most booru engines.
Reasons:
So:
If an API is required, gallery-dl will refuse to scrape HTML as a fallback.
✔ Possible, but only for some booru clones.
Some Gelbooru-style sites expose direct image URLs in HTML
pages.
For these, gallery-dl can scrape HTML because
those sites define two extractors:
gelbooru (API)gelbooru_v01 (HTML legacy)But for Danbooru/Moebooru/Philomena, no such dual implementation exists.
So:
People write custom scripts that:
This is the closest to “function like a browser.”
These behave exactly like a browser:
These behave like:
“Fetch HTML → parse links → download images.”
You could make:
gallery-dl -i -This works because gallery-dl accepts a URL list of direct image URLs without needing an API.
So the wrapper becomes:
“HTML scraper → produce raw image links → feed to gallery-dl.”
| Question | Answer |
|---|---|
| Can gallery-dl act like a browser instead of using APIs? | Sometimes, but not for Danbooru/Moebooru/Philomena. |
| Is there a forced “HTML mode”? | No. Not for Danbooru-type engines. |
| Can another tool scrape HTML like a browser? | Yes — wget, httrack, Selenium, Playwright, Requests-HTML. |
| Can I wrap gallery-dl to use HTML scraping? | Yes — by scraping HTML externally and feeding the extracted URLs to gallery-dl. |
If you want, I can tell you which method is easiest depending on the exact site you're trying to access (Danbooru, Gelbooru clone, Sankaku, etc.), without writing any code.
M: i can access the site without login; how would the api recognize things well enough to know it was automated? (i'm asking cuz i've already included the delay item just out of site courtesy, so it's a bit of a kicker that that same courtesy had them avoid functionality which would allow me to exercise that courtesy.) okay, let's try sending you out to see if you can find a script that does what i'm asking in terms of wrapping gallery-dl, using bash for preference if that's an option. we'll move on to getting you to code without twisting yourself into knots if there's nothing acceptable to be found.
G: Good news: yes, you can build a bash wrapper for entity["software", "gallery-dl", 0] that does what you’re asking (scrape HTML for links, then hand them to gallery-dl). I didn’t find a ready-made, maintained script with exactly your spec (Danbooru-style tag exclusion + zero-pad filenames + delay etc) that I’d trust without tweaks.
So:
tags=xenosaga+-pencil)gallery-dl -i -)If you like, I can locate one good starter script (bash or python) that’s close and then we can adapt it to your needs.
M: find a bash script so we can look at it, please.
G: I searched, and I
couldn’t reliably locate a ready-made bash wrapper script that
meets exactly your spec (HTML-scrape → pipe URLs →
gallery-dl) with clear maintenance.
Here’s a summary:
gallery-dl in one
step.curl/wget +
grep/sed → feed to
gallery-dl -i -).If you like, I can dig deeper (GitHub, GitLab, forums) and try to locate one lesser-known script that is very close — then you and I can adapt it together.
M: let's leave that for a bit. a bash wrapper that gets page code, builds a link list by using a specific tag sequence to identify the page-progression code then proceeding to build the list page by page, then goes through each individual page to get the DL information to feed to gallery-dl - that seems simultaneously like a complex and boring project. instead, answer a minor question - here's the current state of gal.json, which i poked around at - i took out the filename thing entirely, and it think that's it, but i'm not sure. anyway, if i wanted to make it so that files all got popped out beginning with the information from the... i don't know the accurate term, but the test output gives json files containing the fields 'no' and 'filename' that i want to use for the file titles. can i get some syntax, please?
G: Here is the clean, direct answer about how to build filenames from the two fields you mentioned:
nofilename(and optionally still append the extension)
All of this is based on the config you uploaded: fileciteturn1file0
No overthinking — just exact syntax and what works in gallery-dl's formatter.
no)(typical for 4chan and many booru clones)
Use:
"filename": "{no}.{extension}"Zero-padding:
"filename": "{no:0>9}.{extension}"This means:
"no": 12345,000012345.jpg(using the site’s provided filename field)
Syntax:
"filename": "{no:0>9}_{filename}.{extension}"Examples:
000012345_my_image.png000000987_scan0003.jpgno or filename,use fallbacks:
"filename": "{no|filename|id|index:0>9}.{extension}"Interpreter logic:
nofilenameidindex
counterThis guarantees you never get
None.ext.
In your uploaded config, insert this right under
"archive":
"filename": "{no:0>9}_{filename}.{extension}",Or whichever variant you choose.
Example inserted:
"archive": "~/Downloads/gallery-dl/archive.sqlite3",
"filename": "{no:0>9}_{filename}.{extension}",
"#": "Default filename: site/post-or-thread/NNN.ext with zero padding",That’s it.
If unsure, run:
gallery-dl -K "URL"This prints all available metadata fields for that extractor —
including no, filename, id,
etc.
You already verified no and filename appear in
the JSON sidecar, so using them is valid.
filename in gallery-dl does NOT include the
extension.
So your pattern must always append {extension}
yourself:
Correct:
{filename}.{extension}
Incorrect:
{filename}
(because the file would be saved without .jpg,
.png, .webm, etc.)
"filename": "{no:0>9}.{extension}""filename": "{no:0>9}_{filename}.{extension}""filename": "{no|filename|id|index:0>9}.{extension}"If you tell me which one of these you want as your final default, I’ll drop it cleanly into your current config without touching anything else.
M: do you have any idea when the danbooru-style sites started being jealous with their api? because i know it's been a bit since i used gallery-dl, but i didn't think it was that long ago... also, please look around online and see if there's any indication that the api stuff has been getting used to ban users. i don't mind the idea of signing in if it becomes a good idea for a project or whatever - but i don't want to piss off the operators, i'd rather work on workarounds.
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: reasoning_recap; content_type=reasoning_recap]
G: Short version:
Details:
So:
You probably last used gallery-dl in the era where:
That shifted noticeably in the early–mid 2020s, and particularly hard in the last couple of years.
This is even more “per-site & recent”:
api-key +
user-id, giving exactly the
AuthRequired: 'api-key' & 'user-id' needed to access the API ('Missing authentication. Go to api.rule34.xxx for more information')
error you’re seeing. citeturn1search10turn1search2turn1search6So your impression is right:
It was easier recently, and some of these sites have
only just flipped the “API auth mandatory” bit in the last year or
so.
Two separate pieces:
/posts/123/posts.json, /posts.xml, etc.Your 2-second sleep is good manners, but they’re reacting to the global behavior of every bot and scraper, not just you.
The shift to “you must authenticate to use the API” is basically them saying:
“If you’re going to automate, do it where we can rate-limit and attribute it cleanly.”
What I can see:
I haven’t found reputable reports of:
What does show up:
So the reality looks like:
Given your goals:
If you truly don’t want to touch their API at all, HTML-scraping workarounds are an option, but from the operator’s point of view that’s usually less polite, not more — HTML is heavier and harder to control than an authenticated API with rate limits.
Bottom line:
M: Okay, it must have been around August or September that I last used it, because it wasn't around 2013. And I go from site to site, so... For all I know, this new one... I'll go back and check and see if the old one still works. It could just be that it's the new one, anyway. Okay, let's expand thoughts, though. If I just want to sledgehammer my way through the problem, I really like GalleryDL in terms of its flexibility and capabilities. If I did want to take a sledgehammer to the problem by shifting utilities, you gave me a list before, given my setup and preference for command lines and scriptability. What would be the best choices for me as a second tool?
G: Here’s the clean, no-nonsense answer, tuned specifically to your preferences:
I’ll break it into three categories, and then give you the “if you only want one alternate tool, pick THIS” recommendation.
These tools pull HTML and extract links like a browser would, with
zero API dependency.
They’re the closest match to “I want something that does its own
parsing and doesn’t ask permission.”
Strength: ancient, solid, scriptable, handles
pagination, cookies, user agents.
Weakness: crude; can easily miss JS-generated
content.
Used like:
wget --recursive --accept jpg,png,gif,webm --no-parent URL
Great for sites where images are just
<img src="..."> or clickable links in HTML.
Strength: full website mirroring; understands
relative paths, link rewriting, pagination.
Weakness: heavyweight; can overshoot and download more
than you want.
httrack 'https://example.com/gallery/' -O output -%v
Good when the site is purely HTML with straightforward link relationships.
A newer, faster HTML link crawler designed for pentesting but useful
for gathering links.
Strong CLI, pipes well into gallery-dl.
hakrawler -url https://site/page | grep -E '\.(jpg|png|gif|webm)$' | gallery-dl -i -
Strength: fast, good for collecting URLs without
API.
Weakness: no tag logic, no metadata.
These are for when HTML needs actual parsing logic — like extracting a specific tag, finding a class name, or stepping pagination links.
Think: jq, but for HTML.
This is the closest to “browser-ish parsing” while staying fully
CLI-native.
Example:
curl -s URL | pup 'img[src] attr{src}'
You can then pipe into gallery-dl:
curl -s URL | pup 'img[src] attr{src}' | gallery-dl -i -
Strength: extremely scriptable, stays in bash, works
exactly how you think.
Weakness: won’t handle JS-heavy sites.
Lighter than pup.
Uses CSS selectors:
htmlq -a src img < page.html
If you ever decide to go slightly beyond bash, MechanicalSoup behaves roughly like a minimalist, headless browser without all the Selenium overhead.
Strength: easily follows forms, pagination,
cookies.
Weakness: not strictly bash, but extremely
lightweight.
If a site uses JavaScript or blocks HTML-only scraping, these tools let you fetch DOM-rendered content exactly like a browser would.
You said you prefer CLI/scriptability, so I’m excluding GUI-first tools and going directly to headless/controllable ones:
The most stable option in 2025.
You can write scripts that:
<img> or <a>
URLsBut it requires a Python or Node wrapper; not pure bash.
Similar, but heavier and more finicky.
Here’s why:
(no Python, no browser engines, no headless Chrome)
You can cut-and-paste it into tiny one-liners or larger scripts.
You can parse out “next page” links and loop over them.
Meaning: gallery-dl still handles:
Your bash wrapper just supplies the image URLs.
Pup is tiny, maintained, and does not rely on APIs.
This is for when:
It is the sledgehammer that always works — but heavier to install.
Example:
hakrawler -url https://example.com | grep -E '\.(jpg|png|gif|webm)$' | gallery-dl -i -
You already have gallery-dl set up nicely. You want another
tool to “smash through API restrictions” by feeding gallery-dl direct
URLs.
You prefer bash and command-line scriptability.
Therefore:
curl + pup as your secondary
tool.It gives you:
And if the site is fussy or JS-heavy:
If you want, I can design:
Just tell me which direction you’d like.
M: I might try each of your three, but here's the question. Again, Linux setup Fedora. That second part is important because that seems to be kind of the RC Cola to other Linux builds, Coke and Pepsi. Ease of installation, uninstallation, and program size. I don't mind trying them out and then letting them sit until the next time I have to wipe my entire system as long as it isn't going to create a situation where every update involves five different forks or half a gig of file size accidentally.
G: Got it. You want the simplest, smallest, cleanest-install tools that won’t:
And you’re on Fedora, which is indeed the RC Cola of the Linux world: absolutely solid, but many packages default to “developer-friendly over lightweight.”
So here’s the straight, practical breakdown:
These add no heavy dependencies, uninstall cleanly, and won’t mutate over time.
pupFedora often doesn’t ship pup in official repos. So the
safest method is:
cd ~/bin
curl -LO https://github.com/ericchiang/pup/releases/download/v0.4.0/pup-linux-amd64
mv pup-linux-amd64 pup
chmod +x pup
That’s it.
No root required, no dependencies.
If you don’t want it anymore: delete ~/bin/pup.
Verdict:
This is the cleanest, safest choice of all the
tools.
hakrawler~/binInstallation same style as pup:
cd ~/bin
curl -LO https://github.com/hakluke/hakrawler/releases/latest/download/hakrawler_linux_amd64
mv hakrawler_linux_amd64 hakrawler
chmod +x hakrawler
Verdict:
Still extremely lightweight—safe to install and forget.
Perfect for “grab every link beginning with X.”
wget
(recursive)Fedora already has it installed.
Verdict:
Already there; just use it.
These install from Fedora repos and add minimal dependencies.
htmlqsudo dnf install htmlqsudo dnf remove htmlqVerdict:
Still lightweight; Fedora packaging for Rust is very clean.
This is the real sledgehammer.
Installing Playwright on Fedora pulls Chromium, Firefox, and
WebKit binaries, and the Python wrapper has a large dependency
tree.
On Fedora, the command:
pip install playwright
playwright install
Downloads multiple full browsers, which you probably don’t want.
Verdict:
Use only if a site is fully JS-rendered and nothing else works.
You want:
So here’s the clear verdict:
These four tools are:
(due to size, complexity, and messy uninstall on Fedora)
If you want to explore them in order of “least risk → most power”:
All are under 10MB and won't pollute your system.
If you like, I can walk you through installing all four in a
dedicated folder, so they never touch system paths and
can be removed with one rm -r.
M: back trying to gallery-dl to do what i want, on a new ubuntu install. can you give me a local json that does what i've been asking you to do here? or are the tokens old enough that we need to do a refresher?
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: reasoning_recap; content_type=reasoning_recap]
G: Yes — I have enough to give you a fresh local config without a full refresher.
This should match the stable goals you’ve been building toward:
_webm and mp4no first, then filename, then
id, then index so filenames do not collapse to
None.extgallery-dl supports JSON config files,
sleep, archive, filename formatting, and
metadata postprocessors; it also supports YAML/TOML alternatives, but
this version is plain JSON.
citeturn129075search0turn129075search3turn129075search9
Use this as a local gal.json:
{
"extractor": {
"base-directory": "~/Downloads/gallery-dl",
"path-restrict": {
" ": "_",
"/": "_",
"\\": "_",
"|": "_",
":": "_",
"*": "_",
"?": "_",
"\"": "_",
"<": "_",
">": "_"
},
"path-remove": "\u0000-\u001f\u007f",
"path-replace": "_",
"sleep": 2.0,
"sleep-request": 0.0,
"sleep-extractor": 0.0,
"archive": "~/Downloads/gallery-dl/archive.sqlite3",
"directory": ["{category}", "{title|id}"],
"filename": "{no:0>9}_{filename}.{extension}"
},
"format": {
"file": {
"extensions": ["webm", "mp4"]
}
},
"postprocessors": [
{ "name": "metadata", "mode": "json", "event": "after" },
{ "name": "metadata", "mode": "tags", "event": "after" }
]
}A couple of important notes:
Your previous exec / exiftool block is the
part that was blowing up. The formatter error came from that command
template, so I left it out here. The built-in metadata postprocessor is
documented and is the safe part to keep.
citeturn129075search0turn129075search6
If you want the filename to be more fault-tolerant across mixed
sites, use this instead of the filename line above:
"filename": "{no|filename|id|index}.{extension}"or, if you want the leading-zero rule only when no
exists but still avoid None.ext, use:
"filename": "{no:0>9}_{filename|id|index}.{extension}"The reason I’m splitting those is that gallery-dl’s formatter supports fallback chains, but formatting mixed text-and-number fallbacks can get ugly if you try to force zero-padding onto a non-numeric fallback. The examples and discussions around filename formatting and duplicate-file behavior back up being conservative here. citeturn129075search2turn129075search11
Run it locally with:
gallery-dl -c ./gal.json -i list.txtAnd if you want to check the exact available fields for a site before locking the filename pattern in, use:
gallery-dl -K "URL"That is still the quickest way to verify whether a given extractor
exposes no, filename, id, or
something else. citeturn129075search9turn129075search17
The one thing I’m not fully certain about without seeing your current
target site is whether no is always present there. If it
isn’t, I’d use the second, safer filename line instead of the first.
M: this is for just video files, yes? what would i need to either add or remove to download all media?
G: [Non-text content: thoughts; content_type=thoughts]
G: [Non-text content: reasoning_recap; content_type=reasoning_recap]
G: Yes. In that config, this part is what limits it to video files:
"format": {
"file": {
"extensions": ["webm", "mp4"]
}
}If you want all media, remove that entire block.
So change from:
{
"extractor": {
...
},
"format": {
"file": {
"extensions": ["webm", "mp4"]
}
},
"postprocessors": [
...
]
}to:
{
"extractor": {
...
},
"postprocessors": [
...
]
}That will let gallery-dl download whatever media the extractor normally exposes: images, gifs, videos, whatever the site provides.
If later you want a mixed but selective set instead, you can use something like:
"extensions": ["jpg", "jpeg", "png", "gif", "webm", "mp4"]But for everything, the answer is just:
remove the format.file.extensions block
entirely.
M: thought so but had to check. ... i'll prolly be back at some point. but thank you for getting it together.
G: Fair call—that was the only piece gating it.
What you’ve got now is basically a clean, stable baseline config:
None.extThat’s about as “set it and forget it” as gallery-dl gets without going into site-specific tuning.
If you come back later, the next likely pressure points will be:
But for now, you’re in a good place.
M: oh, yes - what is the config specifying flag? -c?
G: Yes — it’s:
-cgallery-dl -c gal.json URLor with your list:
gallery-dl -c gal.json -i list.txt--configExample:
gallery-dl --config gal.json -i list.txt-c, gallery-dl
looks for configs in default locations like:
~/.config/gallery-dl/config.json-c, it uses only
that file (it does not merge with defaults).That’s it — no hidden quirks beyond that.