Simultaneous Schur Decomposition Problem

Consider two matrices $A$ and $B$ which in the absence of error have the same Schur vectors; i.e., there is a $Y \in {\cal O}(n)$ such that $Y^*AY$ and $Y^*BY$ are both block upper triangular where ${\cal O}(n)$ is the set of $n$ by $n$ orthogonal matrices. Now suppose that $A$ and $B$ are somewhat noisy from measurement errors or some other kind of lossy filtering. In that case the $Y$ that upper triangularizes $A$ might not upper triangularize $B$ as well. How does one find the best $Y$?

This is a problem that was presented to us by Schilders [396], who phrased it as a least squares minimization of $F(Y) = \frac{1}{2} (\vert\vert\mbox{low}(Y^*AY)\vert\vert _F^2 + \vert\vert\mbox{low}(Y^*BY)\vert\vert _F^2 )$, where $\mbox{low}(M)$ is a mask returning the block lower triangular part of $M$, where $M$ is broken up into $2 \times 2$ blocks.

For this problem the differential is a bit tricky and its derivation instructive:

$$\begin{eqnarray*} \tr(V^* dF(Y)) &=& \tr( \mbox{low}(Y^*AY)^*\mbox{low}(Y^*AV + ... ... \\ [1.5pt] && + BY\mbox{low}(Y^*BY)^*+B^*Y\mbox{low}(Y^*BY), \end{eqnarray*}$$

where the second equation results from observing that $\tr(\mbox{low}(M)^*\mbox{low}(N)) = \tr(\mbox{low}(M)^*N),$ and the third from properties of the trace.

With second derivatives given by

$$\begin{eqnarray*} \frac{d}{dt} dF(Y(t))\vert _{t=0} &=& AH\mbox{low}(Y^*AY)^*+A... ...Y)}{dt}\right)^*+B^*Y\mbox{low}\left(\frac{d(Y^*BY)}{dt}\right), \end{eqnarray*}$$

where $\dot{Y}(0) = H$, $\frac{d(Y^*AY)}{dt} \vert _{t=0} = H^*AY + Y^*AH$ and $\frac{d(Y^*BY)}{dt} \vert _{t=0} = H^*BY + Y^*BH$.