Forecasting exchange rates is difficult because financial time-series data is too complicated to analyze. In traditional financial studies, economic models and statistic approaches were widely used for predicting exchange rates. Recently, machine learning and deep learning techniques have played increasingly important roles in financial technology studies. This study adopts a deep learning technique called relation networks (RNs) to predict the exchange rates of fiat currencies and cryptocurrencies. To discover the relationship among different currencies, the concept of visual question answering (VQA) is applied in RNs. We also propose a specially designed architecture for the feature extraction stage to consider both spatial and temporal relationships simultaneously. The experimental results show that the proposed approach can achieve higher prediction performance for cryptocurrencies with approximately 65% accuracy rate. We aim to improve traditional approaches and construct a model using the concept of VQA based on RNs to optimize the prediction performance between fiat currencies and cryptocurrencies.