Real-Time Rock Paper Scissors Recognition

Overview

The goal was a working game rather than a high score on a benchmark: a webcam watches two players, finds both hands in the same frame, classifies each gesture, and updates the scoreboard in real time.

The project deliberately starts from a classical, interpretable pipeline (segmentation, morphology, contours and hand-crafted shape descriptors) and only then introduces more expressive models as a comparison. That ordering makes it possible to see exactly where information is lost, and why.

Data and the domain gap

The starting point was the public Roboflow rock paper scissors dataset: 300 × 300 images on a white background, with three balanced classes, 2,520 for training and 372 for testing. It removes most of the difficulty of the background, which is ideal for a first iteration and misleading as a measure of real-world performance.

To close that gap, the team captured 1,362 webcam images of their own (454 per class), with the arm visible, varied angles and lighting, and uncontrolled backgrounds. After a stratified 80/20 split, 1,089 were added to training (3,609 samples in total) and the remaining 273 joined the original test set to form a combined test of 645 images, much closer to how the demo is actually used.

Classical pipeline

1. 01Colour spaceRGB to HSV, Gaussian smoothing
2. 02SegmentationOtsu threshold on the saturation channel
3. 03MorphologyLargest component, hole filling, opening
4. 04Descriptors21 geometric features
5. 05ClassifierRules, SVM or CNN

Otsu’s threshold was implemented by hand on the saturation channel, which separates skin from a white background well. The mask is then cleaned by keeping the largest connected component, filling holes and applying a morphological opening. A second segmenter, a skin-colour range in HSV, was added for variable backgrounds.

From the clean mask, 21 descriptors summarise the shape of the hand: area, perimeter, bounding box, aspect ratio, extent, solidity, compactness, equivalent diameter, prominent peaks, convexity defects, spatial occupancy and Hu moments.

Three families of classifier were compared on top of this:

• a rule-based heuristic baseline;
• SVMs with an RBF kernel on Z-score-normalised descriptors;
• a CNN with four convolutional blocks trained directly on the binary masks, skipping the descriptors entirely.

Results

Each jump in accuracy maps to a specific technical change:

Model	Segmentation	Accuracy	F1 rock	F1 paper	F1 scissors
Rule-based baseline	Otsu	65.1%	0.975	0.000	0.656
SVM, original data	Skin HSV	82.5%	0.842	0.839	0.791
CNN, combined data	Otsu	98.1%	0.996	0.976	0.971

Results on the original test set.

The baseline shows that shape alone carries signal, but it never recognises paper. An SVM on the same descriptors reaches about 75%, and switching to skin-colour segmentation lifts the best SVM to 82.5%.

The decisive step was the webcam data, and the test that matters is the combined one:

Model	Original test	Combined test
SVM, combined data	90.6%	75.7%
CNN, combined data	98.1%	91.0%
CNN with MediaPipe input	—	90.8% (macro F1 90.7%)

The SVM looks strong on the clean data and drops sharply on real scenes; the CNN holds up. When the arm is in view, the silhouette compacts and the geometric descriptors confuse scissors with rock, while the CNN can still use the overall pattern of the fingers and the gaps between them.

Robust hand detection with MediaPipe

Segmentation remained the bottleneck of every shape-based method: when the mask is wrong, the descriptors are wrong, and classification starts from an error. Colour-based segmenters only work in conditions close to the dataset’s.

For the demo, hand localisation moved to MediaPipe HandLandmarker, which runs two small networks: a palm detector that finds each hand, and a regressor that places 21 landmarks on it. Because it depends far less on colour, it keeps the hand of interest in scenes where Otsu and the skin-colour range latch onto the background.

The two-player demo

A single frame contains both players. Each detected hand is cropped into its own 250 × 250 binary mask, classified by the CNN, and the two predictions decide the round. The interface shows the scoreboard, the camera view with detections, both masks and the predicted gesture on each side, so the processing stays visible rather than hidden behind a label.

The work is reproducible end to end: scripts build the combined dataset, train the models and run the full evaluation, with metrics saved as JSON and models serialised, which made the effect of each change visible to the whole team.

Conclusions

• The classical pipeline was enough to demonstrate the course concepts and to justify each technical decision, and it made the limits of descriptor-based classification measurable.
• Segmentation is the bottleneck once the background is no longer controlled.
• Domain data mattered as much as the model: without the webcam samples, good results on a white background meant little.
• MediaPipe for localisation plus a CNN on masks produced a playable demo with about 91% accuracy on real scenes.

Natural next steps would be ensemble models, deeper per-class error analysis and a smoother real-time demo.