bart-large-summary-map-reduce

A text2text model to "map-reduce" summaries of a chunked long document into one.

An explanation of this model's role as a post-processor for textsum (or any other long-doc summarization method similar to the below):

modified flowchart from Google's blog here

Details

This model is a fine-tuned version of facebook/bart-large on the pszemraj/summary-map-reduce dataset. It achieves the following results on the evaluation set:

• Loss: 0.7894
• Num Input Tokens Seen: 14258488

usage

[!TIP] BART supports several speedups for inference on GPU, including flash-attention2 and torch SDPA

an example of aggregating summaries from chunks of a long document:

import torch
from transformers import pipeline

pipe = pipeline(
    "text2text-generation",
    model="pszemraj/bart-large-summary-map-reduce",
    device_map="auto",
)

# examples
text = """"Sangers on a Train" is a 1950 film about a train driver, Guy Haines, who discovers his wife, Miriam, has been murdered in Metcalf, Washington, DC. The film delves into the relationship between Guy and Anne Burton, focusing on Guy's desire for Anne to marry him.
"Screentalk" is a comedy about Anne Burton and her husband, Guy Haines, who are investigating the murder of their daughter, Miriam. The plot revolves around Anne's relationship with Bruno, who has been arrested for his wife's murder. In the second set, Guy and Anne meet at a tennis court in Washington, DC, where they plan to play against each other. Hennessy and Hammond investigate the crime scene, leading to Guy's arrest.
"The Announcer's Boom Forest Hills" is a tennis game between Guy Haines and Bruno Antony, with the score six-five. In the second set, Haines leads three games to four, but his opponent, Bernard Reynolds, attacks him in the third set. Meanwhile, Anne Hennessy and Barbara Hammond are preparing for dinner at the amusement park, where Guy has been waiting for hours. A police car arrives, followed by a taxi. The boatman and detectives follow Guy through the queue, leading to the conclusion that Guy was the man responsible for the accident."""

text = """A computer implemented method of generating a syntactic object. The method includes the steps of providing a plurality of input data sets, each input data set comprising one or more words, wherein each word is associated with at least one non-adjacent second word; creating an exocentric relationship between the first and second words by applying a neo-ian event semantics to the input data in such a way that the neo-antagonistic effect results in the generation of the syntactic object; and storing the generated syntactic object for future use.
    A method of learning and using language is disclosed. The method includes the steps of creating a lexicon of words, wherein each word in the lexicon has at least two possible states, selecting a set of one or more of the possible states of the lexicon to be used as a base state for a subsequent computational operation, and applying the computational operation to the base state to form a new output state.
    A computer implemented method for changing a first workspace to a second workspace. The method includes the steps of creating a new workspace by merging the first workspace with the second workspace, wherein the merging is based on at least one of: an impenetrable condition; a constraint on movement; and a resource restriction.
    The brain is constantly loosing neurons because you doesn't want all the junk around."""

# generate
if torch.cuda.is_available():
    torch.cuda.empty_cache()
res = pipe(
    text,
    max_new_tokens=512, # increase up to 1024 if needed
    num_beams=4,
    early_stopping=True,
    truncation=True,
)
print(res[0]["generated_text"])

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 17868
• gradient_accumulation_steps: 16
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 3.0