Deep learning methods have shown to outperform model-based methods at image upscaling, and exhibit state-of-the-art performance. However, deep learning techniques suffer from drawbacks such as immense data requirements, computational costs, lack of interpretability/explainability, and overfitting. In an attempt to address these issues, the compatibility of algorithm unrolling and self-supervised learning is explored. First, the consequences of utilising the SSL framework data2vec to train a network inspired by ISTA-Net is examined. Then, replacing the linear projection in the encoder of vision transformers with LISTA and pre-training using the masked autoencoder framework, is investigated. Results for the first approach indicate that pre-training an ISTA-Net network using data2vec, might lead to increased generality in scarcely annotated scenarios, however strict attributions remain impossible. The results for the second approach indicate an increased performance in all the experiments. However, to solidify this result, an ablation study on how much can be attributed to the unrolled algorithm over an increased parameterisation, must be conducted.