Next step: Make it Expression<>-oriented.

Making your own math tools is tons of fun and practical!

Since you're asking about potential improvements, my suggestion is to move toward Expression<>-oriented coding next.

First, you define:

public abstract partial class Expression<T>
{
    public T Evaluate()
    {
        return this.Internal_Evaluate();
    }
    protected abstract T Internal_Evaluate();
}

You can ignore the .Evaluate()/.Internal_Evaluate() distinction for now, though I'd suggest that you include it as it may make your life easier later.

Anyway, then you can define stuff like constants

public partial class ConstantExpression<T>
    :        Expression<T>
{
    protected T ConstantValue { get; private set; }

    //  Empty constructor that no external code should ever use:
    protected ConstantExpression() { }

    //  Primary factory-style constructor.
    //  If you add overloads, try to make them call this one,
    //  such that this is the only method that ever includes
    //  "new ConstantExpression<>()" anywhere in your code.
    public static ConstantExpression<T> New(
                T constantValue
        )
    {
        var toReturn = new ConstantExpression<T>();

        toReturn.ConstantValue = constantValue;

        System.Threading.Thread.MemmoryBarrier();  // Just always include this until you have a reason not to.

        return toReturn;
    }

    protected override T Internal_Evaluate()
    {
        return this.ConstantValue;
    }
}

and addition

public partial class AdditionExpression
    :        Expression<double>
{
    protected Expression<double> Argument0Expression { get; private set; }
    protected Expression<double> Argument1Expression { get; private set; }

    //  Empty constructor that no external code should ever use:
    protected AdditionExpression() { }

    //  Primary factory-style constructor.
    //  If you add overloads, try to make them call this one,
    //  such that this is the only method that ever includes
    //  "new AdditionExpression()" anywhere in your code.
    public static AdditionExpression New(
                Expression<double> argument0Expression
            ,   Expression<double> argument1Expression
        )
    {
        if (argument0Expression == null || argument1Expression == null)
        {
            throw new Exception();  // replace with your preferred debug-tracing style
        }

        var toReturn = new AdditionExpression();

        toReturn.Argument0Expression = argument0Expression;
        toReturn.Argument1Expression = argument1Expression;

        System.Threading.Thread.MemmoryBarrier();  // Just always include this until you have a reason not to.

        return toReturn;
    }

    protected override double Internal_Evaluate()
    {
        var argument0 = this.Argument0Expression.Evaluate();
        var argument1 = this.Argument1Expression.Evaluate();

        return argument0 + argument1;
    }
}

with usability helpers like

partial class Expression<T>
{
    public static implicit operator Expression<T>(
                T constantValue
        )
    {
        return ConstantExpression<T>.New(constantValue);
    }

    public static Expression<double> operator +(
                Expression<double> addend0Expression
            ,   Expression<double> addend1Expression
        )
    {
        return AdditionExpression.New(
                   addend0Expression
               ,   addend1Expression
            );
    }
}

Then now that you've got the basic outline for Expression<>'s, you can rewrite your matrix code:

public partial class MatrixExpression
    :        Expression<double[,]>
{
    protected Expression<double>[,] MatrixElementsExpressions { get; private set; }

    protected MatrixExpression() { }
    public static MatrixExpression New(
                Expression<double[,]> matrixElementsExpressions
        )
    {
        var toReturn = new MatrixExpression();

        toReturn.MatrixElementsExpressions = matrixElementsExpressions;

        System.Threading.Thread.MemoryBarrier();

        return toReturn;
    }

    protected override double[,] Internal_Evaluate()
    {
        var matrixElementsExpressions = this.MatrixElementsExpressions;

        var length_0 = matrixElementsExpressions.GetLength(0);
        var length_1 = matrixElementsExpressions.GetLength(1);

        var toReturn = new double[length_0, length_1];

        for (long i_0 = 0; i_0 < length_0; ++i_0)
        {
            for (long i_1 = 0; i_1 < length_1; ++i_1)
            {
                toReturn[i_0, i_1] = matrixElementsExpressions[i_0, i_1].Evaluate();
            }
        }

        return toReturn;
    }
}

Then, it might be tempting to add, say, a .Transpose() method to MatrixExpresion– but don't!

Instead:

public static Expression<double[,]> Transpose(
            this Expression<double[,]> matrixExpression
    )
{
    if (matrixExpression == null)
    {
        throw new Exception();    //  Replace with your preferred error-handling system.
    }

    var toReturn = TransposedMatrixExpression.New(
                matrixExpression
        );

    return toReturn;
}

public partial class TransposedMatrixExpression
    :        Expression<double[,]>
{
    protected Expression<double[,]> MatrixExpression { get; private set; }

    protected TransposedMatrixExpression() { }
    public static TransposedMatrixExpression New(
                Expression<double[,]> matrixExpression
        )
    {
        var toReturn = new TransposedMatrixExpression();

        toReturn.MatrixExpression = matrixExpression;

        System.Threading.Thread.MemoryBarrier();

        return toReturn;
    }

    protected override double[,] Internal_Evaluate()
    {
        var matrixExpression = this.MatrixExpression;

        var matrix = matrixExpression.Evaluate();

        var length_0 = matrix.GetLength(0);
        var length_1 = matrix.GetLength(1);

        var toReturn = new double[length_1, length_0];

        for (long i_0 = 0; i_0 < length_0; ++i_0)
        {
            for (long i_1 = 0; i_1 < length_1; ++i_1)
            {
                toReturn[i_1, i_0] = matrix[i_0, i_1];
            }
        }

        return toReturn;
    }
}

In general, keep Expression<>-definitions slim. New operations shouldn't be additional methods within other classes, but rather each get its own Expression<>, e.g. as we effectively added a .Transpose() method via the class TransposedMatrixExpression above.

Note: Classes and methods are the same thing.

This may be confusing, so I'm quote-boxing it out: you can ignore this point if it doesn't make sense.

C# methods and C# classes are logically equivalent, if we ignore some variation in presentation and implied implementation details. To better understand this, you might look into how anonymous C# methods get their own C# class in the runtime.

Once you understand their equivalence, it'll help frame why we define methods as classes, e.g. as with .Transpose() above.

Tips

1. Put operator definitions, e.g. +, into a partial class Expression<>{ } block, as we did for +(Expression<double>, Expression<double>) above.

2. My coding style may look verbose. I've left a lot of room for things that I suspect most people will want to add as they develop a project like this. I'd advise against trying to make it shorter for a long time, until you get several steps beyond this.

3. It may feel weird to have ConstantExpression<T>'s getting trivially .Evaluate()'d to the T .ConstantValue that they wrap. You may feel inclined to try to reduce overhead by, for example, defining a variant of AdditionExpression that works on a T and an Expression<T>, rather than two Expression<T>'s, to help reduce unnecessary method calls. If you feel strongly about trying this, then it can be a good learning experience – but, it's a mistake.

Next steps.

Obviously, there's a lot to play with here. That's a lot of fun!

Then you can also do stuff like:

1. Creating graphical interfaces for Expression<>'s.

   • I originally did this with WPF. I'd suggest coding it purely in C#; ignore the XML interface.

2. Add in symbolic logic.

3. Extend the logic beyond math into general programming structures.

4. Add in calculus, differential equations, etc..

5. Hoist the whole thing onto a custom evaluation engine.

   • At first, stick with just having an Expression<>-tree doing depth-first .Evaluate()-ing, as above.