Let \(X\) and \(Y\) are two vectors relative to the basis of unprimed system and are represented by \(X'\) and \(Y'\) relative to the basis of a primed system, so that
\begin{equation}
X=PX'; \quad Y=PY' \tag{2.4.12}
\end{equation}
where \(P\) is transformation matrix. Suppose \(X\) and \(Y\) are themselves related by the transformation matrix \(A\) in the unprimed system, so that
\begin{equation}
X=AY\tag{2.4.13}
\end{equation}
\begin{align}
X' \amp = P^{-1}X = P^{-1}AY = P^{-1}APY' \equiv BY' \tag{2.4.14}\\
\text{where} \quad B \amp = P^{-1}AP \tag{2.4.15}
\end{align}
This implies that
\begin{equation}
A = PBP^{-1}.\tag{2.4.16}
\end{equation}
i.e., the matrix of transformation for the same two vectors in the coordinate system is related by
\(P^{-1}AP\) and
\(PBP^{-1}\text{.}\) Hence
\(A\) and
\(B\) are said to be related under similarity transformation. From eqn.
(2.4.15), we have
\begin{equation*}
Tr(B)=\sum\limits_{i}B_{ii}=\sum\limits_{i}(P^{-1}AP)_{ii} =\sum\limits_{i}\sum\limits_{jk}P^{-1}_{ij}A_{jk}P_{ki}
\end{equation*}
\begin{equation*}
= \sum\limits_{j}\sum\limits_{k}\left[\sum\limits_{i}P_{ki}P^{-1}_{ii}\right]A_{jk}
\end{equation*}
\begin{equation*}
\left[\because \quad \sum\limits_{i}(AB)_{ij}=\sum\limits_{i}\sum\limits_{j}a_{ij}b_{ji}\right]
\end{equation*}
after changing the sign for the places.
\begin{equation*}
=\sum\limits_{j}\sum\limits_{k}\left[\sum\limits_{i}PP^{-1}\right]_{kj}A_{jk} = \sum\limits_{j}\left[APP^{-1}\right]_{jj}= \sum\limits_{j}A_{jj} = Tr(A)
\end{equation*}
i.e., trace is invarient under similarity transformation. Also,
\begin{equation*}
|B| = det B = [P^{-1}AP| = |P^{-1}||A||P| = A
\end{equation*}
[\(\because |P^{-1}||P| = 1\)] proved.