Hugging Face's logo

joelmire
/
llama3.1-8b-it-ssf-classifier

PEFT
Safetensors
English
lora
multi-label-classification
Model card Files
xet
Community

Acknowledge Intended Uses & Limitations to access the model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

I have read and acknowledge the intended uses & limitations of the model.

Log in or Sign Up to review the conditions and access this model content.

llama-3.1-8b-it-ssf_classifier

Model description

SSF-Classifier is part of an implementation of the SocialStoryFrames formalism, which is intended to study storytelling practices and reader response on social media, e.g., perceived intent, causal explanation, affective responses.

SSF-Classifier performs multi-label classification on structured inferences generated by SSF-Generator. The label space is defined by the subdimensions in our taxonomy of reader response, mapped across 10 dimensions:

Taxonomy Dimensions

  • Overall Goal: The communicative intent of the comment or post within the broader conversation
  • Narrative Intent: The purpose of the storytelling within the post or comment
  • Author Emotional Response: The emotional state the author would experience while or after telling their story
  • Character Appraisal: Reader judgments of the narrator or other characters' actions or state
  • Causal Explanation: Explanatory inferences readers make to understand aspects of the story
  • Prediction: Predictions readers make about future states or actions in the world of the story
  • Stance: The reader's position or overall opinion in response to a main idea, argument, or point advocated for by the author
  • Moral: The moral values or themes highlighted by the story (based on Schwartz's value theory)
  • Narrative Feeling: Affective responses evoked in readers by the narrative content (characters' situations and events)
  • Aesthetic Feeling: Aesthetic responses evoked in readers by the narrative form, techniques, or style

Subdimensions

  • overall_goal_labels:

    • request_info_support: request factual info or advice about approaches, strategies, etc.
    • provide_info_support: provide facts or advice about approaches, strategies, etc.
    • request_emotional_support: express emotions or characterize a situation to elicit others' comfort, understanding, or empathy
    • provide_emotional_support: provide emotional support to someone by acknowledging their identity, values, or accomplishments; or offering emotional comfort, understanding, or empathy
    • affirm_identity_self: reinforce or assert their own identity, values, or accomplishments
    • provide_experiential_accounts: share a personal story or experience to inform or engage others
    • persuade_debate: advocate for a viewpoint or make an argument to convince others
    • entertain: share an enjoyable or funny post

  • narrative_intent_labels:

    • show_identity: demonstrate an aspect of their identity in action, by example
    • justify_challenge_offer_belief_norm: explain how they came to hold or question a belief or social norm, or to reinforce/demonstrate why it is correct, beneficial, misguided, or harmful; teach a life lesson or influence the behaviors or attitudes of readers
    • entertain: share an enjoyable or funny story
    • release_pent_up_emotions: express themself as a means of emotional release or sensemaking
    • convey_emotional_support_need: describe events that have left them in an unfortunate or unresolved state, in need of emotional support
    • convey_similar_experience: share a story similar to another story or related to a situation under discussion
    • clarify_what_transpired: correct misunderstandings or add missing details associated with an event or situation under discussion

  • author_emotional_response_labels:

    • fear: a response to perceived danger or threat
    • guilt: remorse for violating personal or social standards
    • anger: a strong reaction to perceived harm, injustice, or frustration
    • sadness: a sense of loss, disappointment, or helplessness
    • joy: a state of happiness and contentment
    • pride: a sense of satisfaction from achievements or qualities
    • relief: a release from stress or tension after resolving a concern
    • hope: an optimistic expectation for a positive outcome
    • compassion: concern for others' suffering
    • appreciation: recognition and enjoyment of the good qualities of person, place, or thing
    • connection: closeness or shared understanding with a person, place, or thing

  • character_appraisal_labels:

    • positive_appraisal_narr: a positive judgment of the narrator's actions or state
    • negative_appraisal_narr: a negative judgment of the narrator's actions or state
    • neutral_appraisal_narr: a neutral appraisal of the narrator's actions or state
    • positive_appraisal_other_char: a positive judgment of a non-narrator character's actions or state
    • negative_appraisal_other_char: a negative judgment of a non-narrator character's actions or state
    • neutral_appraisal_other_narr: a neutral appraisal of a non-narrator character's actions or state

  • causal_explanation_labels:

    • narr_explained_by_narr: explaining some aspect of the narrator (e.g., their feelings or behavior) based on the perceived state (e.g., underlying values, beliefs, or goals) or actions of the narrator
    • narr_explained_by_other_char_or_thing: explaining some aspect of the narrator (e.g., their feelings or behavior) based on (1) the perceived state (e.g., underlying values, beliefs, or goals) or actions of a non-narrator character or (2) the perceived state of affairs or event not directly attributed to any character
    • other_char_or_thing_explained_by_narr: explaining some aspect of (1) a non-narrator character (e.g., their feelings or behavior) or (2) the perceived state of affairs or event not directly attributed to any character based on the perceived state (e.g., underlying values, beliefs, or goals) or actions of the narrator
    • other_char_or_thing_explained_by_other_char_or_thing: explaining some aspect of (1) a non-narrator character (e.g., their feelings or behavior) or (2) the perceived state of affairs or event not directly attributed to any character based on (a) the perceived state (e.g., underlying values, beliefs, or goals) or actions of a non-narrator character or (b) some other perceived state of affairs or event not directly attributed to any character

  • prediction_labels:

    • narr_future_state: a prediction about the future state of the narrator
    • narr_future_action: a prediction about the future actions of the narrator
    • other_char_future_state: a prediction about the future state of a non-narrator character
    • other_char_future_action: a prediction about the future actions of a non-narrator character
    • non_char_thing_future_event: a prediction about a future event not directly caused by a character

  • stance_labels:

    • support_belief_norm: a stance that mostly agrees with the stance most recently expressed (explicitly or implicitly) in the preceding conversation
    • counter_belief_norm: a stance that mostly disagrees with the stance most recently expressed (explicitly or implicitly) in the preceding conversation
    • neutral_belief_norm: a stance that is neutral to the stance most recently expressed (explicitly or implicitly) in the preceding conversation

  • moral_labels: (aggregated into higher-level categories following Schwartz's value theory for validation and analyses)

    • self_enhancement:
      • achievement: personal success through demonstrating competence according to social standards
      • power: control or dominance over people and resources; social status and prestige
    • openness_to_change:
      • stimulation: excitement, novelty, and challenge in life
      • self-direction: independent thought and action, expressed in choosing, creating and exploring
    • conservation:
      • security: safety, harmony, and stability of society, of relationships, and of self
      • conformity: restraint of actions, inclinations, and impulses likely to upset or harm others and violate social expectations or norms in everyday interactions, usually with close others
      • tradition: respect, commitment, and acceptance of the customs and ideas that one's culture or religion provides
    • self_transcendence:
      • universalism: understanding, appreciation, tolerance, and protection for the welfare of all people and for nature
      • benevolence: preserving and enhancing the welfare of those with whom one is in frequent personal contact (the 'in-group')
    • hedonism:
      • hedonism: pleasure, enjoyment, or sensuous gratification for oneself

  • narrative_feeling_labels:

    • fear: a response to perceived danger or threat
    • anger: a strong reaction to perceived harm, injustice, or frustration
    • sadness: a sense of loss, disappointment, or helplessness
    • disgust: a reaction of revulsion to something perceived as offensive or repellent
    • joy: a state of happiness and contentment
    • pride: a sense of satisfaction from achievements or qualities
    • hope: an optimistic expectation for a positive outcome
    • compassion: sympathy and concern for others' suffering
    • appreciation: recognition and enjoyment of the good qualities of person, place, or thing
    • connection: closeness or shared understanding with a person, place, or thing

  • aesthetic_feeling_labels:

    • suspense: a feeling of excitement or anxiety in anticipation of an imminent event
    • curiosity: desire for information from the past to explain the present
    • surprise: experiencing an unexpected or shocking event
    • attention_engagement: finding the story to be compelling and capable of holding one's attention, as opposed to mundane or banal
    • transportation: immersion, absorption, or feeling pulled into the world of the story
    • evocation: visualization, e.g., due to vivid language
    • amusement: finding the story to be funny or to contain amusing elements
    • other: an aesthetic feeling not covered by the provided categories

See the paper for more details: (TODO: link).

Typical Workflow

SSF-Classifier is designed to work in a two-stage pipeline:

  1. SSF-Generator given a story and its conversational context, generate free-text inferences about reader response (e.g., overall goal inference: "Many readers from this subreddit would think that the author's overall goal in posting/commenting this text was to clarify misconceptions about whether food delivery services require contracts with restaurants based on their personal experience.")
  2. SSF-Classifier maps those inferences onto fine-grained taxonomy labels (e.g., ["provide_info_support", "persuade_debate", "provide_experiential_accounts])

This provides both information-dense natural language descriptions and structured categorical labels for downstream analysis.

How to Use

This model uses dimension-specific prompt templates that properly format the inference text for classification. We strongly recommend using the prompt builder utilities from the SocialStoryFrames GitHub repository.

GitHub Repository: https://github.com/joel-mire/social-story-frames

In the repo, see main_demo.ipynb to see how to prepare the prompts and run SSF-Classifier.

Intended uses & limitations

This model's primary purpose is to classify inferences generated by SSF-Generator onto the taxonomy's structured label space.

In general, SSF-Classifier and the broader SocialStoryFrames work it partially comprises is designed for English language online conversations and may not generalize well to communities requiring specialized domain knowledge, highly polarized groups, or contexts with idiosyncratic reader reactions. The underlying corpus on which SSF-Generator was trained excludes extremely toxic or sexually explicit content, which may reduce robustness on such inputs and skew judgments toward more positive reactions.

See the paper (TODO: link) for additional details about the intended use for the framework, including SSF-Classifier, as well as important limitations.

Training and evaluation data

Training procedure

SSF-Classifier is trained via LoRA SFT distillation of GPT-4.1 on meta-llama/Meta-Llama-3.1-8B-Instruct. The model was trained on the train split of joelmire/ssf-corpus.

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 25
  • distributed_type: multi-GPU
  • num_devices: 3
  • total_train_batch_size: 24
  • total_eval_batch_size: 24
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 3.0

Training results

We evaluated SSF-Classifier through expert human annotations (N=1,764 annotations). We evaluated both the GPT-4.1 reference classification outputs and SSF-Classifier's direct outputs.

Across the ten dimensions of reader response in our taxonomy, SSF-Classifier achieved:

  • Average Micro F-1: 0.848 (min: 0.65)
  • Average Macro F-1: 0.791 (min: 0.62)

SSF-Classifier exceeds, matches, or is within 0.05 F-1 points of GPT-4.1 for a majority of taxonomy dimensions (7/10 for Micro F-1, 6/10 for Macro F-1), with all dimensions being within 0.1 points from their GPT-4.1 counterparts.

For low-level training results, see training_loss.png and train_results.json.

Framework versions

  • PEFT 0.15.1
  • Transformers 4.51.3
  • Pytorch 2.5.1+cu124
  • Datasets 3.1.0
  • Tokenizers 0.21.1

Related Resources

License

This model is derived from meta-llama/Meta-Llama-3.1-8B-Instruct and is subject to the Llama3.1 community license.

Citation

TODO

Downloads last month
2
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for joelmire/llama3.1-8b-it-ssf-classifier

Base model

meta-llama/Llama-3.1-8B
Finetuned
meta-llama/Llama-3.1-8B-Instruct
Adapter
(1322)
this model

Dataset used to train joelmire/llama3.1-8b-it-ssf-classifier

Evaluation results