DOA.
SELECTED PROJECTS

Multimodal Activity Recognition.

For my MSc dissertation, I implemented and evaluated a pipeline that combines 3D body pose with wearable sensor data to recognise physical activity. I owned the workflow from preparing the data to comparing the models.

Individual MSc dissertation / University of Plymouth / September 2025My role: data preparation, modelling and evaluationPython / PyTorch / MM-Fit

Can two sources of movement data work better together?

A body pose describes how someone is positioned. Wearable inertial measurement units (IMUs) record acceleration and rotation over time. My research question was whether combining these signals could recognise fitness activities more effectively than using either one alone.

I used the existing MM-Fit dataset, covering 21 recording sessions, to implement a fusion model and compare it with pose-only and IMU-only models.

Original dissertation flowchart connecting MM-Fit, preprocessing, single-modality and fusion models, training and evaluation
My original project pipeline, dissertation Figure 6 (PDF page 45). It shows the preprocessing and comparison workflow for pose, IMU and fusion models.

What I owned

This was my individual dissertation. I developed the data-processing and modelling workflow in Python and PyTorch, prepared the experimental inputs and evaluated the classifiers.

  • Data preparation: I aligned pose and IMU sequences, prepared labelled windows and handled differences in scale and input structure.
  • Model implementation: I built the pose CNN, the inertial LSTM and the classifier combining their learned features.
  • Experimental comparison: I evaluated fusion against the single-modality models using classification metrics and confusion matrices.
  • Technical interpretation: I examined generalisation and wearable-inference constraints, and documented what the experiments could support.

Different signals, one activity interval

The first challenge was alignment. Pose and inertial recordings have different sampling rates and structures. Combining unrelated windows would make the fusion experiment difficult to interpret, even if both branches trained successfully.

I prepared synchronised windows so the two inputs described the same activity interval. Pose windows contained 150 frames with 18 joints and three coordinates per joint; inertial windows contained 250 samples across 24 channels.

Three-dimensional skeleton plot showing the pose input representation
Pose visualisation reproduced in dissertation Figure 7 (PDF page 47), attributed in the report to Strömbäck, Huang and Radu (2020).
Multichannel IMU signal plots with red, blue and green time-series traces
IMU visualisation reproduced in dissertation Figure 8 (PDF page 48), attributed in the report to Strömbäck, Huang and Radu (2020).

These figures show the two input types used in the research. Select a figure to view it at full size.

I also needed to handle missing pose observations, differences in input scale and class imbalance. My preprocessing used pose interpolation, hip-centred pose normalisation and inertial-channel standardisation. For training, I used non-activity downsampling, weighted cross-entropy and label smoothing.

These decisions connected the data preparation to the modelling problem: preserving movement information while limiting the influence of scale differences and dominant classes.

The pipeline I implemented

Original dissertation architecture showing pose convolutions, a bidirectional inertial branch, attention, feature concatenation and activity classification
Architecture as documented in my dissertation, Figure 9 (PDF page 53). The original diagram and its labels are retained.
  1. Prepare paired examples. I aligned the modalities, normalised their inputs and created labelled activity windows.
  2. Learn from each modality. I used a CNN for the pose representation and an LSTM for temporal patterns in the inertial sequences.
  3. Fuse the features. I concatenated the two learned representations and used a fully connected classifier to predict the activity.
  4. Compare and inspect. I evaluated the fused model alongside the pose-only and IMU-only models, inspecting accuracy, precision, recall, F1 scores and confusion matrices.

I used NumPy, SciPy and scikit-learn alongside PyTorch for data processing and evaluation.

What the experiments showed

The comparison reported in my dissertation favoured the fused model over the pose-only and IMU-only models. Within those experiments, combining posture and inertial movement information was more effective than relying on a single modality.

Performance also varied between the evaluation groups. That made generalisation an important part of interpreting the result: strong performance on one set of recordings does not establish reliable performance for new users or environments.

The outcome was an implemented and evaluated multimodal workflow, with comparisons that made the contribution of each input visible. This finding concerns my experiments on MM-Fit. Results from the original researchers provided benchmark context and are separate from my own results.

What this work developed

I strengthened my ability to take heterogeneous time-series data through preprocessing, model implementation and evaluation. The project required me to connect decisions about windows, normalisation and class balance to the behaviour of the resulting classifiers.

I also developed a more careful approach to reporting model performance. Class-level precision, recall and macro F1 help explain behaviour that an overall accuracy figure can hide, especially when activities are unevenly represented.

What I would improve next

  • Cross-user evaluation: I would make session and participant splits explicit in a saved experiment manifest, then repeat comparisons across held-out users.
  • Missing-modality testing: I would test sensor disconnection and unreliable pose observations to understand how the fusion model behaves when an input deteriorates.
  • Physical-device measurement: I would measure latency, memory use and energy consumption on the intended device. The project did not establish these through a physical wearable deployment.

The study focused on structured fitness activity. Exercise monitoring and rehabilitation are possible directions for further investigation; the project was not a clinically evaluated healthcare system.

Project context

Individual MSc Artificial Intelligence dissertation, University of Plymouth, September 2025. Research on CNN-LSTM fusion of 3D pose and IMU data using MM-Fit.

The full academic report remains private. This case study presents my implementation, experimental findings and development priorities.