Mutually orthogonal unitary and orthogonal matrices

We introduce the concept of order-d n-OU and n-OO sets, which consist of n mutually orthogonal order-d unitary and real orthogonal matrices under Hilbert-Schmidt inner product. We show that for arbitrary d, there exists order-d d²-OU set. However, real orthogonal matrices show strict limits, as we prove that an order-three n-OO set exists only if n≤4. As an application in quantum information theory, we establish that the maximum number of unextendible maximally entangled bases within a real two-qutrit system is four. Further, we propose a new matrix decomposition approach, defining an n-OU (resp. n-OO) decomposition for a matrix as a linear combination of n matrices from an n-OU (resp. n-OO) set. We show that any order-d matrix has a d-OU decomposition. As contrast, we prove the existence of real matrices that do not possess any n-OO decomposition by providing explicit criteria for an order-three real matrix to have an n-OO decomposition.