Breaking CAPTCHA repository

This repository contains the report and code for our project in Scalable Machine Learning and Deep Learning (ID2223) at KTH.

The aim of the project is to break CAPTCHAs using deep learning technologies. Initially we focus on simple CAPTCHAs to evaluate the performance and move into more complex CAPTCHAs. The training dataset is generated from an open source CAPTCHA generation software. Tensorflow is used to create, train and test the network.

Generating the dataset

The datasets is generated using a Java based CAPTCHA generator. The easiest way is to import the Java project into IntelliJ, compile, and execute breakingCAPTCHA/data_gen/src/kth/id2223/Main.java. We have generated the following datasets.

Description	Size	Training samples	Test samples
Digits only	38 MB	9502	100
Digits and characters	197 MB	49796	100
Digits and characters with rotation	39 MB	10000	100
Digits and characters with rotation	198 MB	49782	500
Digits and characters with rotation	777 MB	196926	500

Each dataset contains jpeg images containing a CAPTCHA with five characters. The characters are lowercase letters (a-z) or numbers (0-9). We used the fonts "Arial" and "Courier" with and without rotation. An example of the created CAPTCHAs is displayed below. Our intention was to mimic the CAPTCHAs created by Microsoft.

Simple digit-only CAPTCHA
Characters and digits without rotation
Characters and digits with rotation

Executing the CNN

We have developed the code using Tensorflow 8.0 with Python 2.7.6 and Nvidia GPUs with CUDA 8.0. The CNN can be executed from the root folder of the project with cd simple_CNN and PYTHONPATH=".." python CAPTCHA_cnn.py.

After a successful training, the learned model is saved as model.ckpt. Then you can use PYTHONPATH=".." python restore_CAPTCHA_cnn.py filename.jpg to predict an image file using the learned model.

Results

First unsuccessful tries

CNN with three conv. layers and two fully connected layers accuracy of CAPTCHAs with five digits or lowercase letters without rotation. Training in 100 batches and 10000 training samples.

Digit-only CAPTCHAs

CNN with three conv. layers and two fully connected layers accuracy of CAPTCHAs with five digits without rotation. Training in 157 batches, 39250 training samples, and testing with 100 CAPTCHAs.

Digit and letter CAPTCHAs

CNN with three conv. layers and two fully connected layers accuracy of CAPTCHAs with five digits or lowercase letters without rotation. Training in 199 batches, 49750 training samples, and testing with 500 CAPTCHAs.

Digit and letter CAPTCHAs with rotation

CNN with three conv. layers and two fully connected layers accuracy of CAPTCHAs with five digits or lowercase letters with rotation. Training in 787 batches, 196926 training samples, and testing with 500 CAPTCHAs.

Examples of correct and false predictions

Correct
prediction: 54563
prediction: grh56
prediction: fb2x4

False
prediction: 82298
prediction: k76ap
prediction: fffgr

Report

We have compiled a full blog-style report with more details. Selected results can be found in the results folder.