LETTERNET: A CNN based Encoder/Decoder for Visual Data Transfer beyond Barcodes

Machine-readable static data can be encoded in a variety of ways. While barcodes, such as QR codes, are a popular form, human writing utilizing OCR can also be used, but this is not optimal if the goal is machine readability. Thus, the question arises, what form of representation would a machine learning algorithm invent to be able to transmit data over a static medium such as paper? In this work, we will contribute a CNN that learns to generate an image from characters in such a way that it can be decoded back to the original characters by a decoder under given degradations. These three components, encoder, degradations, and decoder, are trained in an end-to-end fashion. In the evaluation, we show how the generated images appear for different degradations, and also show that this form is more robust than QR codes decoded by ZBar for different string lengths. Furthermore, we show that even ZBar with prior removal of blur and noise by a state-of-the-art method performs worse than the proposed LETTERNET. Finally, our sim-to-real tests show that the sim-to-real gap is small even though the training is based on purely artificial datasets and degradations.