As high-level vision tasks are increasingly used in face-related fields, face image restoration (FIR) has also received much attention. Despite significant advances in this field, there are issues that have been overlooked. First, we discover that most FIR works utilize geometry or reference facial priors to help restore realistic face images. However, estimated priors from low-quality images show inaccurate results, leading to decreased restoration performance. Also, exploiting facial priors requires additional process and computational costs. Second, we argue that existing FIR approaches take the evaluation metrics widely used in general image restoration to measure the performance. However, existing evaluation metrics do not correlate well with human judgment. The absence of an appropriate evaluation metric for the FIR metric could become one of the major bottlenecks in FIR algorithm development. In this study, to overcome the limitations mentioned above in FIR, we first apply knowledge distillation to propose an efficient model that does not require additional priors. Second, we introduce a perceptual no-reference metric designed for face image quality assessment.