FSCOCO comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offering both object- and scene-level abstraction.

FSCOCO Dataset


Abstract

We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. With practical applications in mind, we collect sketches that convey well scene content but can be sketched within a few minutes by a person with any sketching skills. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offering both object- and scene-level abstraction. Each sketch is augmented with its text description. Using our dataset, we study for the first time the problem of fine-grained image retrieval from freehand scene sketches and sketch captions. We draw insights on: (i) Scene salience encoded in sketches using the strokes temporal order; (ii) Performance comparison of image retrieval from a scene sketch and an image caption; (iii) Complementarity of information in sketches and image captions, as well as the potential benefit of combining the two modalities. In addition, we extend a popular vector sketch LSTM-based encoder to handle sketches with larger complexity than was supported by previous work. Namely, we propose a hierarchical sketch decoder, which we leverage at a sketch-specific “pre-text” task. Our dataset enables for the first time research on freehand scene sketch understanding and its practical applications

Dataset Statistics

For our dataset, we compute two estimates of the category distribution across our data: (1) Upper Bound: based on semantic segmentation labels in images and (2) Lower Bound: based on the occurrence of a word in a sketch caption.

Total Sketches # Categories # Categories per Sketch # Sketches per Category
Mean Std Min Max Mean Std Min Max
10,000 92/150 1.37/7.17 0.57/3.27 1/1 5/25 99.42/413.18 172.88/973.59 1/1 866/6789

Dataset Sample and Comparison with existing dataset.

Sample Comparison FSCOCO dataset

License / Terms of Use

The dataset is under review. Please send an email to mail@pinakinathc.me to get the password to unzip the dataset. You are not allowed to share this dataset. All rights are reserved. We shall release this dataset under Creative Commons Attribution 4.0 License once this dataset is published.

How to cite this dataset

@article{fscoco,
    title={FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context.}
    author={Chowdhury, Pinaki Nath and Sain, Aneeshan and Gryaditskaya, Yulia and Bhunia, Ayan Kumar and Xiang, Tao and Song, Yi-Zhe},
    journal={arXiv preprint arXiv:2203.02113},
    year={2022}
}

Download this dataset

Since this dataset is under review, you might need a password to unzip the dataset. Please send an email to mail@pinakinathc.me. Downloading this dataset means you agree not to share this dataset until the dataset is under review.

Important Note: Due to social media ban, we have decided NOT to release the dataset unzip password until mid of 2022. I am maintaining a list of all people sending request for an early sneak peak. I shall send the password to these people immediately after the ban is lifted.

Acknowledgements

This dataset would not be possible without the support of the following wonderful people:

Anran Qi, Yue Zhong, Lan Yang, Dongliang Chang, Ling Luo, Ayan Das, Zhiyu Qu, Yixiao Zheng, Ruolin Yang