Cognitive distortions are negatively biased thinking patterns and erroneous self-statements resulting from and leading to logical errors in one’s own internal reasoning. Cognitive distortions have an adverse effect on mental health and can lead to mental health disorders in extreme cases. This paper belongs to a bigger project which aims to provide an application for detecting and classifying cognitive distortions in texts. As no public data sets were available for the task, the first contribution of the proposed work lies in providing an open-source labeled dataset of 14 cognitive distortions consisting of 34370 entries collected via crowd-sourcing, user questionnaires, and re-purposing emotions dataset from social media. The dataset is collected in cooperation with a licensed psychologist. We implemented a baseline model using Naïve Bayes and Count Vectorizer and different CNN, LSTM, and DNN classifiers to classify cognitive distortions based on the dataset. We investigated the usage of different word embeddings with the best-performing models. The best-performing model relied on a CNN with pre-trained Sentence-BERT embedding with an F1-score of 84 % for classifying cognitive distortions. The best-performing model was built into C-Journal, a free journaling and mood-tracking mobile application that pinpoints potential thinking distortions to the users.