flan-t5-base-paragrapher

This model is designed to preprocess, clean, and reformat text chunks containing line breaks, word breaks, and references into coherent plain text paragraphs. The resulting paragraphs can be used with other models like agentlans/flan-t5-small-title and agentlans/text-summarization.

Model description

The flan-t5-base-paragrapher is a fine-tuned version of google/flan-t5-base, trained on a dataset of open-source introductory social science textbooks. While it was trained on academic texts, it should work well with other types of educational and academic content.

The model achieves the following results on the evaluation set:

• Loss: 1.5175
• Number of Input Tokens Seen: 49 815 380

Intended uses & limitations

This model is intended for preprocessing and reformatting text chunks into coherent paragraphs. It can be particularly useful for:

1. Cleaning up text extracted from PDFs or OCR systems
2. Reformatting text with irregular line breaks or word breaks
3. Preparing text for further processing or analysis

Limitations:

• The model may not perform optimally on highly specialized or technical texts outside its training domain.
• Very long input sequences may be truncated due to the model's maximum sequence length (512 tokens).

Training and evaluation data

The model was trained on a dataset compiled from open-source textbooks. Due to licensing constraints, the specific training data is not published.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• Learning rate: 5e-05
• Train batch size: 8
• Eval batch size: 8
• Seed: 42
• Optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• LR scheduler type: linear
• Number of epochs: 10.0

Training results

Click to expand training results

Framework versions

• Transformers 4.44.2
• PyTorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.19.1

Usage

Here's an example of how to use the model:

from transformers import T5Tokenizer, T5ForConditionalGeneration

# Load the tokenizer and model
tokenizer = T5Tokenizer.from_pretrained("agentlans/flan-t5-base-paragrapher")
model = T5ForConditionalGeneration.from_pretrained(
    "agentlans/flan-t5-base-paragrapher", device_map="auto"
)

# Define input texts
# Note: These aren't real citations. Only for demonstration purpose.
input_texts = [
    """ge with a narrative—whether through books, films, or oral traditions—we are invited into another person's experience (Brown & Thompson, 2023). This immersion allows us to see the world through different perspectives, breaking down barriers of misunderstanding and prejudice. For example, novels like Harper Lee's "To Kill a Mockingbird" challenge readers to confront issues of racism and injustice through the eyes of a child (Williams, 2018). Similarly, contemporary works such as Chimamanda Ngozi Adichie's "Americanah" explore themes of identity and belonging in a globalized world (Nguyen & Roberts, 2020). By sharing these experiences through storytelling, authors can cultivate empathy in their audiences, encouraging them to reflect on their own beliefs and biases.
    Shaping Identity Through Narratives
    Stories also play a crucial role in shaping personal and collective identities. From childhood tales told by parents to the myths and legends that define cultural heritage, narratives help individuals understand their place in the world (Anderson & White, 2021). They provide frameworks thro""",
    """cia, M., & Patel, R. (2022). Cultural insights through literature: A comparative analysis. International Journal of Cultural Studies, 15(3), 201-215. Johnson, L., & Lee, H. (2019). Oral traditions: Preserving culture through storytelling. Anthropology Today Journal, 34(4), 56-60. Kumar, P. (2021). Epic tales: Literature as a reflection of society. Literary Critique Review, 29(1), 34-50. Lee, J., & Martinez, F. (2021). Voices unheard: Marginalized narratives in digital spaces. Journal of Digital Culture Studies, 7(2), 45-67. Martinez, C., & Chen, Y. (2022). Cultural navigation: Identity in a globalized world. Global Studies Review Jou""",
]

# Tokenize input texts
input_ids = tokenizer(
    input_texts, return_tensors="pt", padding=True, truncation=True
).input_ids.to("cuda")

# Generate outputs
outputs = model.generate(input_ids, max_length=512)

# Print generated outputs
for output in outputs:
    print(tokenizer.decode(output, skip_special_tokens=True) + "\n")

Example output:

Through storytelling, we are invited into another person's experience, breaking down barriers of misunderstanding and prejudice. This immersion allows us to see the world through different perspectives, fostering empathy and re-evaluating our own beliefs and biases. For instance, Harper Lee's "To Kill a Mockingbird" challenges readers to confront issues of racism and injustice through the eyes of a child, while contemporary works like Chimamanda Ngozi Adichie's "Americanah" explore themes of identity and belonging in a globalized world. By sharing these experiences through storytelling, authors

The study of cultural insights through literature has yielded valuable insights into the world. Ci and Patel (2022) conducted a comparative analysis of cultural insights through literature, highlighting the importance of cultural storytelling in preserving culture. Kumar (2021) argued that oral traditions can preserve culture through storytelling, highlighting the importance of storytelling in preserving culture. Lee and Martinez (2021) explored marginalized narratives in digital spaces, highlighting the need for cultural navigation in a globalized world. These studies collectively demonstrate the importance of cultural navigation in fostering identity and identity in a globalized world.