Weird Tagger V1

A tagger for "weird" things.

"What's this?"

You thought danbooru is cursed? Tough luck. Pixiv has even more cursed things that don't get surfaced on danbooru. Welcome to the Internet.

While standard anime models (like WD14 or Camie) are trained on densely-captioned, crowdsourced Danbooru posts, this model was trained directly on the untamed wilds of Pixiv. It is designed to recognize 1,802 highly specific, visually distinct tags that traditional models miss. This includes:

• Obscure doujin artists and circles.
• Niche character variants.
• Extreme R-18/R-18G fetishes.
• Compositional quirks (e.g., RPG status screens, UI overlays, specific manga framing).

Model Architecture

A dual-head Vision Transformer based off from camie-tagger-v2 was used. Dual-head meaning that there's an additional new head that is used to predict the "weird tags" mentioned in the above.

Data Filtering

While it's easy to do the actual crawling (A couple hours on a cpu node, no-biggie), filtering for actual useful tags was the harder problem. Our goal for this tagger was basically this:

"Make the model recognise tags that appears on pixiv but are generally avoided to be included in danbooru."

1. Counting

Since each tag is attributed to 1 post (which may have more than 1 image), we basically counted the number of tags for each post

Additionally we skipped posts which are flagged as AI-Generate or has a set of AI tags.

[
    "aiイラスト",
    "nightshade",
    "novelai",
    "nai",
    "midjourney",
    "waifulabs",
    "holaraai",
    "aiart",
    "aiアート",
    "ai生成",
    "stablediffusion",
    "nanobanana",
    "zeta",
    "aiアニメ",
    "comfyui",
    "ai-generated",
    "aigenerated",
    "検索妨害"
]

When counting, we also normalize the tag with the following function:

def normalize(tag_name:str):
    base_tag = (
        # Convert katakana to hiragana
        tag_name.translate(KATA_TO_HIRA)
        # Strip all traditional punctuations
        .translate(DEPUNCT)
        # Clean spaces
        .strip()
        # lower case
        .lower()
        # remove underscores
        .replace("_", "")
        # Remove weird unicode escaped things
        .replace("\xa0", "")
        # Remove touhou prefix and collapse them
        .removeprefix("東方")
    )
    return base_tag

Lastly, we also get a count of authors per tag.

Spice Level

When counting, it would be worthwhile to know how SFW <-> NSFW a tag is. To do this, we compute a spice level which is based on the sum of 2 values (sl (Sanity level, the amount of "skin" shown, from pixiv), xRestrict (User tagged level))

This created a sliding score from 2 (since sanity level's lowest value is 2) all the way to 6.

2. Filtering

Once counting is all done, we have obtained ~8,780,197 "unique" (normalized) tags. The first couple steps to filtering we basically, remove tags with certain prefixes, suffixes or "contains" certain characters. the final tables after reading the tags manually ended up looking like this

MAP_SKIP_ENDSWITH = tuple(
    [
        normalize(i)
        for i in (
            "users入り",
            "user入り",
            "users 入り",
            "user 入り",
            "古シリーズ",
            "ほしい",
            "ほしいい",
            "学マス",
            "補習組",
            "シリーズ",
            "捏造",
            "あるある",
            "ちゃん",
            "イラスト",  # Short handle for illustration
            "のラスト",  # <thing>... last
            "キャラ",  # ... chara
            "あーと",  # <thing>... art
            "の日",  # <thing>... Day? Unsure if i should pull it out or nah...
            "いいのよ",  # Some things...
            "良いのよ",  # Some things...
            "好良い",  # Some things...
            # It's kamen rider... again...
            "らいだー",  # Same as above...
            "すごい",  # ...well...
            "絵日記",
            "のか",
            "てみた",
            "くらい",
            "注意",
        )
    ]
)

MAP_2_SKIP_STARTER = tuple(
    [
        normalize(i)
        for i in (
            "オリジナル",
            "ユート",
            "シリーズ",
            "捏造",
            "仮面らいだー",  # inflation of sfw tags by ~ 200
            "うるとらまん",  # inflation of sfw tags by ~ 75
            "ぷりきゅあ",  # inflation of sfw tags by ~ 80
            "きゅあ",  # inflation of tags by ~ 130. Yes this is the short form for pricure. im sure.
            "イラスト",  # Short handle for illustration
            "なにこれ",  # ...well...
            "何これ",  # ...well...
            "これは",  # ...well...
            "やさしい",  # ...well...
            "絵",  # Odd pick but it does prune ~ 100 tags. most of it is meta tags
            "今日の",
            "今日も",
            "ほぼ日",
            "うちの",
            "どうして",
            "らいだー",
            "毎日",
        )
    ]
)

CONTAINS = re.compile(
    "|".join(
        [
            normalize(i)
            for i in [
                "艦隊",
                "小説",
                "漫画",
                "フルカラー",
                "イラスト",
                "キャラクター",
                "ください",
                "艦これ",
                "誕生",
                "生誕",
                "かわいい",
                "可愛い",
                "お絵描",
                "描き",
                "書き",
                "希望",
                "朝の",
                "二次",
                "せらふ",
                "プロジェクト",
                "クロスオーバー",
                "ノベル",
                "ゲーム",
                "一日",
                "好き",
                "やられ",
                "美しい",
                *["ケット" + str(i) for i in range(1, 10)],
            ]
        ]
    )
)

Next was to remove tags which have only 1 character since these tags tend to generally not give enough infomation about the thing in question. Afterwards, we removed really popular tags (>100,000 posts) and really obscure tags (<100 posts). With that reduction, we wound up with ~189K Tags.

From reading the data, we removed any tags which contained punctuation and any kind of english text. This does remove serieses and franchises but that's a reasonable trade off to make here. Additionally, we aren't after english anyway so--

120K tags... Still too many, but after passing it through danbooru aliases, as well as filtering for these 2:

FOREIGN_REGEX = re.compile(
    r"[\uAC00-\uD7A3\u3130-\u318F"  # Korean Hangul
    r"\u0E00-\u0E7F"  # Thai
    r"\u0600-\u06FF"  # Arabic/Persian
    r"\u0400-\u04FF]"  # Cyrillic
)


EMOJI_REGEX = re.compile(
    r'[\U0001F300-\U0001FAFF\u2600-\u26FF\u2700-\u27BF]'
)

is_unwanted_san() # Basically some "XX-San" things, details ommited for abbreviety.

Down to ~98K.

Spice Level Buckets

Since we computed spice levels bucketting earlier, we can actually bucket tags from "SFW" all the way to "Hell". The buckets used were:

if spf < 2.5:
    # Mostly SFW, some might be fall into lewd.
    agg_d["t_kind"] = "sfw"

elif spf < 3.003:
    # This is mostly suggestive content.
    agg_d["t_kind"] = "lewd"

elif spf < 5.003:
    # "Expect NSFW here"
    # This bin contains tags which are strictly in the grey area of:
    # "Some illust is not porn but, do expect porn here."
    agg_d["t_kind"] = "pseudo-nsfw"

elif spf < 7.003:
    # Well come the the bar, or well...
    # This typically means the tag is mostly NSFW.
    agg_d["t_kind"] = "nsfw"

elif spf < 7.322:
    # Stuff where the author doesn't *exactly*
    # Know how to bin it either in R-18 / R-18G
    agg_d["t_kind"] = "bizare"
else:
    # Guro focussed content
    agg_d["t_kind"] = "hell"

Once bucketted, we saw that tags are mostly in the sfw section.... However, coming back to our original vision:

"...generally avoided to be included in danbooru."

Also includes artists too. So an additional round of filtering tags into either a "general" or "artist" specific tag.

We then selected all tag groups which has < 3000 tags. Some additional japanese based morpheme checks were used as well as a bit more restrictive tag check for >200 posts but by now we filtered the tags to a managable number of around 1823.

The final tags used is uploaded to tags.accum.inverse.txt file for your viewing.

3. Download

Reasonably straight forward, other than the fact that we had to downsample tags with more than 5000 posts to not wash out the under used tags

4. Train!

Again reasonably straightforward, but grafting was not super straight forward. Training on a hyper-sparse, Positive-Unlabeled (PU) dataset (where most tags have fewer than 200 images) required an extreme training pipeline:

• Asymmetric Focal Loss (gamma_pos=0, gamma_neg=4, clip=0.05) was used to silence overwhelming negative background noise of the 1.2M image dataset so the model can actually learn rare tags.
• Augementation: Random Horizontal Flips, Affine Shifts, and Random Resized Cropping. The usual.
• IRFS (Instance Repeat Factor Sampling): was used to oversample rare tags to ensure the model learned it.

While we considered and ran a Logistic Regression AUC probe, we noted that all tags more or less pass the AUC threshold.

Limits

• This is very experimental. I mean, not much else to say man other than YMMV. I dunno what theshold is good even for the weird tags. ~0.5 threshold from casual poking would be correct
• DO NOT USE THIS FOR TAGGING REAL LIFE CONTENT. Basically I cannot guarantee the performance translate well over to something OOD like real life images...
  • If you really want to do this, you shoudl probably train your own or look elsewhere.
• Un-Translated. The tags are not translated at all, basically you're gona read moonrunes. Deal with it. Someone can put through a model and get a reasonable translation. I might create a json map later

AnyRes Patching

Tried AnyRes patching and it does seem to help... Though you might want to fiddle around with it.