← zohaibksyed.com

Financial Headline Sentiment Analysis

[Project Poster — CSCI 4521]

Machine learning to use what the news says to make you money in the markets.

By: Zohaib Syed

The objectiveof this project was to implement machine learning to analyze financial sentiment in text to gain market insight from a source that isn’t as saturated as statistics. The data set that was used in this project contains around 5,000 different phrases and their sentiment collected by researchers in 2014 pursuing a similar objective.

Sentence Transformer + MLP

Raw Text

Financial phrase

Encoder

all-roberta-large-v1 · 1024 dim. embedding

Custom MLP

Layer Norm (1024)

Hidden Layer 1

Linear Layer 1024→256

GELU Activation

Dropout (p=0.5)

Hidden Layer 2

Linear Layer 256→64

GELU Activation

Dropout (p=0.4)

Output

Linear 64→3

Prediction

Positive, Neutral, Negative

LLM Baseline

Raw Text

Financial phrase

Qwen3-VL-2B-Instruct

“You are an expert financial sentiment analyst with 30+ years of experience. Using your expertise, Classify the financial phrase as positive (0), neutral (1), or negative (2). Do not leave anything unclassified. Make your best prediction no matter what. Return only one token: 0, 1, or 2.”

+ 3 labeled few-shot examples → RegEx parse output

Avg response time 0.49 s

Prediction

Positive, Neutral, Negative

Key Takeaway — Per Class F1

MLP (all-roberta-large-v1)Qwen3-VL-2B
00.250.50.751MLP — Macro F1: 0.760.76Qwen — Macro F1: 0.540.54Macro F1MLP — Positive: 0.790.79Qwen — Positive: 0.490.49PositiveMLP — Neutral: 0.850.85Qwen — Neutral: 0.700.70NeutralMLP — Negative: 0.640.64Qwen — Negative: 0.440.44Negative

Qwen avg latency: 0.490 sec/sample · MLP: <1 ms/sample · MLP is ~500× faster

F1 Score for the negative class was significantly worse than others for MLP, but not as bad for Qwen.

Visual Analysis

  • “Looks like its booking a one way ticket to its 40 week MA near 50. Losing 10 week here $LULU http://chart.ly/7xb9h9b”MLP

    Pred: NegativeReal: Neutral

  • “Sold out $TZA 45 $PUT (down $1), which were hedging my 45 $CALL. Letting the calls ride solo now.”MLP

    Pred: NegativeReal: Positive

  • “$SIMO Basing nice. A close over 24.18 is bullish $QQQ $SPY $STUDY watch for a close near HOD. Currently up +3%”Qwen

    Pred: NegativeReal: Positive

Pattern: Both models struggled with phrases containing stock tickers ($LULU, $TZA) and URLs — noise strings that carry no sentiment but appear to bias predictions toward negative. Neither model learned to ignore non-sentiment tokens. This suggests future improvement via pre-processing to strip tickers and URLs before encoding.

Conclusion / Takeaways

  1. MLP dominates across all metrics — +22pt accuracy, +0.22 macro F1 vs Qwen
  2. Weighted cross-entropy loss effectively handles the 11.1% negative class imbalance
  3. Simpler is better — 2 hidden layers outperform deeper networks on 5K samples
  4. Domain-specific encoders (all-roberta-large-v1) are the key performance driver

Recommendations

  1. Deploy the MLP — superior accuracy, near-zero latency, minimal compute cost
  2. Do not deploy an LLM for classification without fine-tuning on domain data. Strip tickers, URLs & hashtags as a pre-processing step to reduce noise
  3. Be wary of accuracy as a metric on imbalanced data — use macro F1