The intrinsic and extrinsic defects in LiNbO3 (LN) bulk crystals have been investigated with experimental and theoretical methods for many years. However, nowadays, scientific attention has turned to nanocrystals (e.g., for quantum nanophotonic applications). Herein, the results of spectroscopic measurements and theoretical calculations on Yb3+- or Er3+-doped LN nanocrystals are presented. A series of congruent LN nanocrystals doped with Er3+ or Yb3+ have been produced by grinding process under wet conditions in a high-energy ball mill. The effect of size reduction on the rare-earth (RE) ions has been followed by absorption measurement of the characteristic electronic transitions using a Fourier-transform infrared spectrometer. Out-diffusion of the RE ions from the nano-LN particle and the appearance of an unordered phase have been concluded from the experimental results. GULP software has been used to make force-field calculations on a Li245Nb245O735 unit to model the nano-LN structure containing Er3+ or Yb3+ dopants. From the calculations taking into account, one Er3+ or Yb3+ ion, their out-diffusion, and the appearance of the ions in the unordered phase have been demonstrated, while in the double-doped case, independent RE incorporation has been found.