Towards Manipulating AI Text Controllably (Proof of Concept LoRA)
Civitai | ☕️ Ko-fi (support my work)
Introduction
Text has been an area in AI image generation that has been its weak point for the longest time. For a while, mangled, unreadable text was the norm; but nowadays we have both frontier models like NanoBanana or GPT Sunburst and local models like MOSS-Image, Qwen-Image-2.1, or Ideogram 4.0 that can handle text perfectly. Yet, even though these models excel at rendering text, none of these models are very good at editing and manipulating text in a precise manner. For example, we can change the word 'Advertising' to 'Design' with local or frontier models, and they would edit it correctly; However, the result is random. You cannot reliably specify if the text is left, center or right aligned, nor can you specify the exact position of the text. Additionally, you can't specify the vertical scale of the text or the size of the text. These are just some of the many properties that aren't very controllable when it comes to text in AI image generation. The LoRA FontBox is an attempt to bridge this gap.
FontBox allows you to manipulate the following features of text:
- Edit text and specify left, right, or center alignment
- Specify the exact size of text
- Specify the exact position of text
- Turn text faux bold or faux italic while maintaining the font family
- Adjust vertical scale of text
We utilize FLUX.3 Image Klein as the model of choice when making this LoRA because it supports bounding box (bbox) and .json editing, which is crucial for this to work. While FLUX.3 Image Klein is not a particularly good model, especially compared to models like MOSS-Image or Qwen-Image-2.1, these bbox and .json features separate it from the pack and are crucial for precice text editing.
Custom Node
Because .json prompting and bbox editing are required inputs for this LoRA, and writing .json and bbox coordinates by hand is a pain, we created a companion custom node for ComfyUI that allows you to adjust and manipulate text in a beautiful UI with good UX. Just write the text you want manipulated, type what you want, and hit Run! Boilerplate prompt templates are baked into the node, so no manual adjustments are needed. Create guides, and enable snapping for precice editing. Workflows and the custom node are located here on Github.
Showcase
Editing Text
With FontBox, you can edit text and specify left, center, or right alignment (one of these must be chosen):
Prompt for left alignment In <ref_image_0>, change the text "CHASE OUR HEARTS!" <obj_2> to "STAY SILLY!" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[52,36,126,960],"desc":"STAY SILLY!"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]'
In <ref_image_0>, change the text "CHASE OUR HEARTS!" <obj_2> to "STAY SILLY!" and right align it, and maintain the exact font typographic style, character spacing, colors, and style. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[52,36,126,960],"desc":"STAY SILLY!"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
In <ref_image_0>, change the text "CHASE OUR HEARTS!" <obj_2> to "STAY SILLY!" and center align it, and maintain the exact font typographic style, character spacing, colors, and style. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[52,36,126,960],"desc":"STAY SILLY!"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
FontBox tries its best adhere to text with perspective or distortion:
Prompt for "Break" In <ref_image_0>, change the text "BRE" <obj_2> to "HEA" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, change the text "EAK" <obj_3> to "ART" and right align it, and maintain the exact font typographic style, character spacing, colors, and style. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[9,0,710,663],"desc":"HEA"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[281,337,833,1000],"desc":"ART"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
text with gradients and dropshadows:
Prompt for "big mas" In <ref_image_0>, change the text "MERRY" <obj_2> to "BIRTH" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, change the text "BIG MAS" <obj_3> to "DAY" and right align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, keep "HAPPY" <obj_4> exactly unchanged. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[362,298,629,916],"desc":"BIRTH"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[634,116,892,870],"desc":"DAY"},{"id":"obj_4","from":"ref_image_0","src_bbox":[107,111,365,755],"tgt_bbox":[107,111,365,755],"desc":"HAPPY"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
In <ref_image_0>, change the text "ハート" <obj_2> to "Patch" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, change the text "レスキューミー" <obj_3> to "Your Heart!" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, keep "SWEET STEADY" <obj_4> exactly unchanged. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[125,219,308,615],"desc":"Patch"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[179,144,486,859],"desc":"Your Heart!"},{"id":"obj_4","from":"ref_image_0","src_bbox":[73,238,134,525],"tgt_bbox":[73,238,134,525],"desc":"SWEET STEADY"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
and tries its best to adhere to text on paths:
Prompt In <ref_image_0>, change the text "whatever happens" <obj_2> to "feeling generous?" and center align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, change the text "just roll with it" <obj_3> to "you can support me!" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. In <ref_image_0>, change the text "funky" <obj_4> to "ko-fi" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[148,83,496,600],"desc":"feeling generous?"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[570,336,818,787],"desc":"you can support me!"},{"id":"obj_4","from":null,"src_bbox":null,"tgt_bbox":[344,548,406,699],"desc":"ko-fi"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
Moving text
With FontBox, you can move entire words or phrases:
Prompt In <ref_image_0>, move the text "CATCH CATCH" <obj_2> from its source box to its target box, and preserve its font family, colors, and all other styling. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":"ref_image_0","src_bbox":[40,49,241,352],"tgt_bbox":[770,684,979,992],"desc":"CATCH CATCH"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
or move individual letters:
Prompt In <ref_image_0>, move the text "O" <obj_2> from its source box to its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, move the text "K" <obj_3> from its source box to its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, move the text "I" <obj_4> from its source box to its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, move the text "A" <obj_5> from its source box to its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, keep "N" <obj_6> exactly unchanged. In <ref_image_0>, The "K" should be behind the subject <obj_7>. In <ref_image_0>, The "O" should be behind the subject <obj_8>. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":"ref_image_0","src_bbox":[67,267,183,466],"tgt_bbox":[115,267,231,466],"desc":"O"},{"id":"obj_3","from":"ref_image_0","src_bbox":[65,467,182,669],"tgt_bbox":[162,470,279,672],"desc":"K"},{"id":"obj_4","from":"ref_image_0","src_bbox":[60,664,183,732],"tgt_bbox":[213,666,335,734],"desc":"I"},{"id":"obj_5","from":"ref_image_0","src_bbox":[65,734,182,948],"tgt_bbox":[258,735,375,949],"desc":"A"},{"id":"obj_6","from":"ref_image_0","src_bbox":[64,46,177,265],"tgt_bbox":[64,46,177,265],"desc":"N"},{"id":"obj_7","from":null,"src_bbox":null,"tgt_bbox":[164,470,279,667],"desc":""},{"id":"obj_8","from":null,"src_bbox":null,"tgt_bbox":[114,267,233,465],"desc":""},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
Adjusting text size
With FontBox, you can draw an extended bounding box to adjust the text size of words:
Prompt In <ref_image_0>, adjust the size of the text "gaeul" <obj_2> to fit its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, change the text "09:02" <obj_3> to "" and left align it, and maintain the exact font typographic style, character spacing, colors, and style. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[123,365,201,543],"desc":"gaeul"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[418,291,633,842],"desc":""},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
or individual letters:
Prompt In <ref_image_0>, adjust the size of the text "R" <obj_2> to fit its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, adjust the size of the text "A" <obj_3> to fit its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, keep "ssimilate" <obj_4> exactly unchanged. In <ref_image_0>, keep "ot" <obj_5> exactly unchanged. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[21,532,230,672],"desc":"R"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[177,271,498,489],"desc":"A"},{"id":"obj_4","from":"ref_image_0","src_bbox":[292,483,489,994],"tgt_bbox":[292,483,489,994],"desc":"ssimilate"},{"id":"obj_5","from":"ref_image_0","src_bbox":[58,660,220,769],"tgt_bbox":[58,660,220,769],"desc":"ot"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
You can also adjust the vertical scale of letters or words by drawing an extended bounding box of the letters:
Prompt In <ref_image_0>, adjust the vertical scale of the text "W" <obj_2> to fit its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, adjust the vertical scale of the text "E" <obj_3> to fit its target box, and preserve its font family, colors, and all other styling. In <ref_image_0>, keep "IS" <obj_4> exactly unchanged. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[517,68,714,185],"desc":"W"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[731,315,931,390],"desc":"E"},{"id":"obj_4","from":"ref_image_0","src_bbox":[725,163,815,283],"tgt_bbox":[725,163,815,283],"desc":"IS"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
Bolding or italicizing text
With FontBox, you can also (faux) bold or (faux) italicize text:
Prompt for "Italicize" In <ref_image_0>, italicize the text "NANANANA" <obj_2>, and preserve its font family, colors, and all other styling. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[211,76,389,928],"desc":"NANANANA"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
In <ref_image_0>, bold the text "Baby Cry For Me" <obj_2>, and preserve its font family, colors, and all other styling. In <ref_image_0>, bold the text "Pale Memories" <obj_3>, and preserve its font family, colors, and all other styling. In <ref_image_0>, bold the text "Anything You Want" <obj_4>, and preserve its font family, colors, and all other styling. Keep the rest of the image completely unchanged, preserving the backdrop, art, and all other text <obj_1>. [{"id":"obj_2","from":null,"src_bbox":null,"tgt_bbox":[674,177,706,365],"desc":"Baby Cry For Me"},{"id":"obj_3","from":null,"src_bbox":null,"tgt_bbox":[715,471,748,637],"desc":"Pale Memories"},{"id":"obj_4","from":null,"src_bbox":null,"tgt_bbox":[761,608,794,814],"desc":"Anything You Want"},{"id":"obj_1","from":"ref_image_0","src_bbox":[0,0,1000,1000],"tgt_bbox":[0,0,1000,1000],"desc":"backdrop"}]
Training Regimen
We utilize the program Affinity to create the dataset. We use various fonts in various styles, place text on paths, and edit text with short and long paragraphs and phrases.
Where it fails
As this is a proof of concept, it is prone to unstability for some generations. FLUX.3 Image Klein is very sensitive to types of typography, and resolution. See below as an example:
As you can see, I instructed the "B" and the "d" to move down, and yet the entire phrase "Beyond" moved down with it, which is incorrect. However, we can correct this discrepancy by adding "keep" markers to the text we want not touched. "Keep" markers is a native FLUX.3 Image command that tells FLUX.3 Image Klein to not touch what is marked. After we add a couple "keep" markers, we get this output:
In order to add a "keep" marker in the FontBox custom node, draw a box and change the mode from "New" to "Keep" in the combobox. Type in what you want to keep, and your set! You should not have to do this all of the time since FontBox usually gets it correct the first time, but this is useful to course-correct uncooperative outputs.
Additionally, FLUX.3 Image Klein is very sensitive to resolution. Use at least 1 megapixel or higher (recommended ~2MP) to maintain specific typographic details in text.
In addition to sensitivity to resolution, FontBox is not designed to manipulate wordmarks or logotypes. Highly stylized wordmarks will likely not work, however your mileage may vary.
Lastly, while FontBox can generalize to different languages, FontBox is trained primarily in English. Your mileage may vary for text manipulation of other languages.