# New linear algebra operations and symbolic features in TMS Analytics & Physics library

Bookmarks:

#### Friday, July 13, 2018

TMS Analytics & Physics developing library supports operations with array and matrix variables and includes special package of linear algebra functions. New version 2.5 introduces many capabilities and improvements for linear algebra operations. First, new operators introduces specially for vector and matrix operations: cross product operator ‘×’, matrix inverse operator ‘`’, norm operator ‘||’. In previous version of Analytics library matrix multiplication with standard linear algebra rules were implemented using multiply operator ‘*’. In new version matrix-matrix and matrix-vector multiplication implemented using the cross product operator, and the multiply operator used for by-element multiplication. This leads to more compliant algebra rules: multiply operation is always commutative, as for numbers as for matrices and vectors, and the cross operation is not commutative. By the same reason, the inverse operation used now for calculating inverse of square matrices, as the division operation now is always by-element one (so, the division operation is the inverse of multiplication). The norm operator evaluates the L2 norm of vectors and matrices, and the absolute operator ‘||’ used for evaluating absolute values of vector and matrix components. Let us consider the difference between multiplication operator and cross operator with the simple example. Suppose there are the following two matrix variables:
```A [3x3] =       		  	B [3x3]=
( 0 0.1 0.2)				( 1 0 0)
( 1 1.1 1.2)      			( 0 2 0)
( 2 2.2 2.3)      			( 0 0 3)
```
Evaluating expressions for multiplying the matrices using cross and multiply operators we get the following results:
```A×B	= [3x3] =            	A*B	= [3x3]=
( 0 0.2 0.6)				( 0   0   0)
( 1 2.2 3.6)				( 0 2.2   0)
( 2 4.4 6.9)				( 0   0 6.9)
```
As can be seen from evaluation result, the first operation implemented with standard linear algebra rules for matrix product (https://en.wikipedia.org/wiki/Matrix_multiplication), as the second operation is just by-element multiplication of the matrices. The latter operation frequently used in signal and image processing algorithms (https://en.wikipedia.org/wiki/Hadamard_product_(matrices)). Combination of both operations can be used in one expression.

In addition to the operators, new functions for matrix arguments have been introduced in Linear Algebra package: trace of a square matrix tr(M), determinant of a square matrix det(M), adjoint of a square matrix adj(M), condition number of a square matrix cond(M), pseudo-inverse of a rectangular matrix pinv(M). The second improvement concerns symbolic capabilities for vectors and matrices. In previous versions, the only way of using arrays in formulae was introducing variables for them. The version 2.5 allows using explicit vector and matrix expressions in any symbolic formula. Example of simple vector expression with four components ‘[sin(x) cos(y) e^-x a*b]’. Matrix expressions are 2D rectangular arrays of expression elements. For example, ‘[[i j k] [sin(a) cos(a) -1]]’ is a 2x3 matrix expression.

Vector and matrix expressions can be used in formulae like vector and matrix variables. The difference is that the former are unnamed but their components can depend on other variables. Thus, we can calculate symbolic derivatives of the expressions. For example, the derivative of the above vector expression by ‘x’ is ‘[cos(x) 0 –(e^-x) 0]’.

For evaluating values of vector and matrix expressions, all their components must have the same type. The component types supported are: Boolean, Real and Complex.

As example of evaluating vector expressions let us consider the following problem: there is the array ‘A’ with four components and we need to calculate new array, selecting components from the ‘A’ by different four conditions. For solving this problem we can use vector form of ‘if’ function. The expression can be the following: ‘if{[a>1 b<3 a-x>0 b+x<2]}(A [-1 -2 -3 -4])’. Here the condition of the ‘if’ function defined by the vector expression with four Boolean expression components, depending on other variables: ‘a>1’, ‘b<3’, ‘a-x>0’ and ‘b+x<2’. The arguments of the functions are array variable ‘A’ and vector expression with four real components. Evaluating the expression with some given variable values, we get the following result:
```x = 1.5
a = 1
b = -2
A = (1 0.5 0.25 0.125 )

if{[a>1 b<3 a-x>0 b+x<2]}(A [-1 -2 -3 -4]) = (-1 0.5 -3 0.125 )
```
The example demonstrated that vector and matrix expressions can be used together with array and matrix variables and, in some cases, they are more flexible, because their components can depend on other variables.

These new improvements of linear algebra capabilities allowed realizing vector basis for linear least squares approximation. Here is the source code of an approximation example using the vector basis:
```var
approximator: TLinearApproximator;
basis: TLinearBasis;
func: string;
parameters: TArray<TVariable>;
exponents, cValues: TArray<TFloat>;
A: TVariable;
dim, order: integer;
begin
// Setting parameters for approximation.
dim:= 1; // One dimension.
// exponent values
exponents:= TArray<TFloat&ht;.Create(0.0, 0.5, -0.5, 0.7, -0.7);
order:= Length(exponents); // Basis order is the size of the array.
// Vector variable for exponent array.
A:= TRealArrayVariable.Create('A', exponents);
parameters:= TArray<TVariable>.Create(A); // Parameters for Vector basis.
func:= 'e^(A*X[0])'; // Vector function for basis - returns real vector.

// Create the basis with the specified function and parameters.
basis:= TLinearVectorBasis.Create('X', 'C', order, dim, parameters, func);

approximator:= TLinearLeastSquares.Create; // Linear approximator.
// Calculate the optimal basis coefficients.
cValues:= approximator.Approximate(basis, vData, fData);

// Using the approximation (basis with optimal coefficients)...
end;
```
As can be seen from the code, approximation basis function has very compact form – ‘e^(A*X[0])’, because it uses vector variable ‘A’ and so, the return value of the function is real vector. The result of the approximation for some given data is the following:

```F(X, C, A) = C•e^(A*X[0])

F(X) = [0.015894261869996 -0.000559709975637631 0.560703382127893
6.67799570997929E-5 -0.60704298180144]•e^([0 0.5 -0.5 0.7 -0.7]*X[0])
```

Advantages of using vector basis are:
• Simple basis function expression (in vector form).
• Easily create high-order basis (using big array parameter).
• Use arbitrary coefficients for the basis (fractional exponents, powers and so on).
• Change the basis without changing the function (use other values of the parameter array ‘A’).
In addition to the vector and matrix expressions, there are some other symbolic features in the version 2.5. One feature is calculating symbolic expression for n-order or mixed derivative with one function call. The code below calculates mixed derivative of a complicated expression:
```var
translator: TTranslator;
F, Dxy: string;
begin
translator:= TTranslator.Create;

F:= '(y-A*ln(y))/x+B*(v(x+y))^(2/Pi)';
Dxy:= translator.Derivative(F, TArray<string>.Create('x','y'));

// using mixed derivative Dxy...
end;```
The output result for the derivative is:
```F(x,y)  = (y-A*ln(y))/x+B*(v(x+y))^(2/Pi)
dF/dxdy = (-1+A/y)/x^2+B*(1/p-1)*(x+y)^(1/p-2)/p
```
Note, that after derivative evaluation the result expression simplified. New simplifications in version 2.5 are: put minus operator under sum and product expressions (first expression in the sum ‘(-1+A/y)/x^2’); simplification of square root and power expressions (second expression in the sum ‘B*(1/p-1)*(x+y)^(1/p-2)/p’). Finally, there is new symbolic function that allows getting all variable names from math expressions. Here is an example of using the function:
```var
F: string;
Names: TArray<string>;
begin
F:= 'A[i]*sin(2*Pi*x)+A[j]*cos(2*Pi*y)+B[i]*e^((1+I)*x)+B[j]*e^((1-2I)*y)';
Names:= TTranslator.VariableNames(F);

// using the Names array...
end;
```
The result is the following:
```F = A[i]*sin(2*Pi*x)+A[j]*cos(2*Pi*y)+B[i]*e^((1+I)*x)+B[j]*e^((1-2I)*y)
Variable names: A, i, x, j, y, B
```
Note, that the function does not include variable names in the result twice (i, j, x, y, A, B). In addition, standard constants, like Pi and Euler number, not included in the names list.

The version 2.5 is already available here. Source code of the example application can be downloaded from here. Other examples of using vector/matrix operations, expressions and vector basis with graphics visualization included in demo projects of the version 2.5.

Bruno Fierens

Bookmarks:

# Statistics extension for TMS Analytics & Physics developing library

Bookmarks:

#### Friday, May 18, 2018

TMS Analytics & Physics developing library includes many extensions for various mathematical applications: complex numbers, linear algebra, special functions, common fractions, numerical methods. New version 2.4 introduces package for statistical analysis. The extension allows evaluating base statistical properties of real data samples (mean value, median and others), generating sequences of special numbers (Fibonacci, prime numbers and many others), creating arithmetic, geometric and harmonic progressions, working with probability distributions.

It should be noted here, that as the Analytics library allows symbolic evaluations of very complicated formulae, some statistical problems can be solved without using the special package. For an example, the Linear Algebra package includes functionality for processing array and matrix data. So, we can solve, for an example, the following problem: find the sum of ‘A’ array elements those are greater than ‘x’. The result can be calculated with the following code:

The output for the code is the following:

The problem is solved with the ‘if’ function that compares all items of the ‘A’ array with ‘x’ value and generates new array which contains elements of ‘A’ if the condition satisfied and 0 value if not. When the sum of the result array calculated we get the solution of stated problem. The goals of new Statistics package is providing the simplest formula for evaluating base statistical characteristics and solving statistical problems that cannot be solved without specialized library. First, let us consider evaluation of base statistical characteristics of some discrete sample values, stored in ‘A’ array. This can be done with the following code:

For this data, we get the following statistical values:

Mean of A = 0.155625
Median of A = 0.135
Mode of A = 0.1
Variance of A = 0.012124609375
Deviation of A = 0.440447215906742

As can be seen from the code above, every value calculated with one function call. The function ‘Mean{P}(A)’ here evaluates the mean value of the data. This function is parametric and the ‘P’ parameter specifies the ‘power’ of the mean (https://en.wikipedia.org/wiki/Mean#Power_mean). The parameter value can be one of the four: ‘-1’ – harmonic mean, ‘0’ – geometric mean, ‘1’ – arithmetic mean, ‘2’ – quadratic mean. In our case P=1 and we calculated the arithmetic mean of ‘A’. One of the advantages of using the symbolic capabilities of the Analytics library is that, even if some feature is not realized in the package directly, one can use the formula for this feature. Let us consider how to do this for other types of ‘mean’ values. There is so called generalized mean (https://en.wikipedia.org/wiki/Mean#Generalized_means) that is defined by the formula:

where f is a function and f-1 is its inverse. For an example, if f is exponent, then f-1 is natural logarithm. The ‘exponential’ mean function is not provided with the statistical package, but it can be easily evaluated with the following simple formula:

which produces the output:

Exponent Mean of A = 0.161840327746035

The value of ‘exponent’ mean is slightly differs from common arithmetic mean that follows from the different meaning of the characteristics.

One of main features of Statistics package is that it allows working with probability distributions and solve real statistical problems (https://en.wikipedia.org/wiki/Probability_density_function). The version supports 6 distributions: Gauss (normal), Laplace, Cauchy, Gumbel, logistic and exponential. Let us consider the following statistical problem: there is a known probability distribution for lifetime of a bacteria species. Let us suppose the distribution is Gaussian function with parameters µ and s (https://en.wikipedia.org/wiki/Normal_distribution). We need to calculate the probability that the bacteria dies on the time interval [t1..t2].
This probability defined by the formula:

where f(x) is the probability density function (PDF) of the distribution. The value of the probability can be evaluated with the following code:

First, we added the variables for the distribution’s parameters to the translator and variables for the time interval. The function 'GaussPDF{µ s}(t1 t2 100)' produced the array of 100 values for the Gaussian PDF with specified parameters (µ=5 hours, s=0.5 hours) on the specified interval of [5.5..6.0] hours. Then we used numerical integration of the values, replacing the integral with the appropriate summation. The answer for our question is - the probability that the bacteria dies on the interval 5.5-6 hours is about 13.6%. The version 2.4 is already available. Source code of the example application for statistics analysis can be downloaded from here.

Bruno Fierens

Bookmarks:

# Improvements for symbolic expressions simplification in TMS Analytics & Physics developing library

Bookmarks:

#### Wednesday, December 20, 2017

One of the unique features of the Analytics library is symbolic derivatives calculation. The derivative can be calculated just with several lines of Delphi code, for an example:

```var
f, df: string;
t: TTranslator;
begin
t:= TTranslator.Create;
f:= 'x^2*e^(x/3)';
df:= t.Derivative(f, 'x');
//...
end;```

The Analytics library guarantees that the result derivative expression is syntactically and mathematically correct and satisfies the derivative rules. From the first version, the derivative algorithm automatically simplifies the result derivative expression. Nevertheless, the library is not a Computer Algebra system, so it cannot simplify all the result expressions to the optimal representation.

New version 2.3 introduces new simplification algorithms for providing more readable and short symbolic derivatives. New simplification algorithms include: constants reducing in sum and product expressions; reducing nested sum or product expressions to one; reducing negation pairs in product expressions and others.

There are some examples of functions and their derivative expressions, evaluated with the old version of the library and with the 2.3 version:

```f(x)       : x^2*e^(x/3)
df/dx (old): (2*x)*e^(x/3)+(e^(x/3)*(1/3))*x^2
df/dx (2.3): 2*x*e^(x/3)+e^(x/3)/3*x^2

f(x)       : 2*x^3+A/4*x^2-x/3
df/dx (old): (3*x^2)*2+((2*x)*A)/4-1/3
df/dx (2.3): 6*x^2+1/2*x*A-1/3

f(x)       : (A/2)*arctan(x^2)
df/dx (old): ((1/(1+(x^2)^2))*(2*x))*(A/2)
df/dx (2.3): x*A/(1+x^4)

f(x)       : P{n-1 m+2}(x)
df/dx (old): ((((n-1)+1)-(m+2))*P{(n-1)+1 m+2}(x) -(((n-1)+1)*x)*P{n-1 m+2}(x))/(x^2-1)
df/dx (2.3): ((n-m-2)*P{n m+2}(x)-n*x*P{n-1 m+2}(x))/(x^2-1)

```

As can be seen from the examples, new version gives shorter and more readable symbolic expressions for different function cases. The simplified representation of the formulae also reduces the calculation time. The new library version is already available here

Bruno Fierens

Bookmarks:

# TMS Analytics for nonlinear multiparametric approximation of surface data

Bookmarks:

#### Thursday, October 05, 2017

The new version 2.2 of TMS Analytics and Physics pack contains the nonlinear least squares approximation tools. The nonlinear approximation tasks are widely used in statistics to fit experimental data with some random distribution function.

Let us consider the example of surface data fitting with the 2D Gaussian function. The Gaussian distribution of 2 variables x and y (https://en.wikipedia.org/wiki/Gaussian_function) has the following math expression:

Except the amplitude A, the distribution nonlinearly depends on four parameters: center coordinates x0, y0 and deviations sx, sy. The task of fitting data with the Gaussian function is to find optimal values of the four parameters, those minimize the error of the distribution and the experimental data zi(xi,yi), i=1..N.

The approximation task can be solved by the numerical tool with the following code:

```var
variables, coefficients: TArray;
f: string;
basis: TNonlinearBasis;
cValues: TArray;
appr: TNonlinearApproximator;
opt: TSolverOptions;
eValues: TArray;
eValue: TFloat;
begin
variables:= TArray.Create('x', 'y'); // 1
coefficients:= TArray.Create('x0', 'Sx', 'y0', 'Sy'); // 2
f:= 'e^-((x-x0)^2/(2*Sx^2)+(y-y0)^2/(2*Sy^2))'; // 3
basis:= TNonlinearScalarBasis.Create(variables, coefficients, nil, f); // 4

appr:= TGaussNewtonLeastSquares.Create; // 5
appr.C0:= TArray.Create(0.1, 0.1, 0.1, 0.1); // 6
opt:= TSolverOptions.Create(true);
opt.MaxIterationCount:= 100;
opt.Precision:= 0.2;
appr.Options:= opt; // 7

cValues:= appr.Approximate(basis, xyData, zData); // 8

// use basis with optimal coefficients
end;
```

Line 1: Create array of variable names.
Line 2: Create array of four distribution coefficient names.
Line 3: Set up the approximation function expression.
Line 4: Create the nonlinear basis instance with specified data.
Line 5: Create the appropriate nonlinear approximator instance.
Line 6: Set up initial guess of coefficient values for nonlinear solution.
Line 7: Set up appropriate nonlinear solution options.
Line 8: Solve the problem (finding optimal values).

When optimal values of the Gauss distribution parameters found, they can be used with the basis instance to calculate the distribution function in any specified point (x,y) or for other analysis methods, like derivative calculations and so on.

On the pictures below there are results of the approximation made with some generated noisy data. The result presented as 1D section of the 2D function for two different x values: x=0.0 and x=0.6.

In this sample, we used the TMS FNC Chart to display the values and the approximated function. The advantage of using the TMS FNC Chart is that it can be used both for VCL and FMX applications and we can use the same code to create a VCL Windows application or a FireMonkey cross platform application.
The source code of the example application can be downloaded here.

The TMS Analytics and Physics pack version 2.2 includes also many other useful numerical tools: linear least squares approximation, 1D and 2D integration, ordinary differential equation solution, function analysis. Due to the addition of array and matrix support in version 2.2, many numerical calculations can be implemented in vector form with small and compact formulae. The version can be downloaded from the product page.

Bruno Fierens

Bookmarks:

# Array and Matrix operations with Analytics 2.2

Bookmarks:

#### Tuesday, September 26, 2017

New version 2.2 of TMS Analytics and Physics pack includes total support of operations with N-element arrays and MxN matrices. The operations realized in the Linear Algebra extension package. Array and matrix operations allow advanced calculations with small, compact formulae.

Let us consider the examples of using the array/matrix operations for solving some common math/statistics problems. First of all we need to add special variables to the translator instance:
```var
av: TArray<TFloat>;
mv: TArray<TArray<TFloat>>;
begin
translator:= TTranslator.Create;

translator.Add('µ', 0.0);

av:=TArray<TFloat>.Create(0.2, -0.1, -0.25, 0.4);
translator.Add('A', av);

av:=TArray<TFloat>.Create(-0.2, 0.2, -0.1, 0.25, 0.35, 0.15, -0.04, 0.01, -0.12, 0.1);
translator.Add('B', av);

SetLength(mv, 4, 3);
mv[0,0]:= 0.5; mv[0,1]:=-0.4; mv[0,2]:= 0.33;
mv[1,0]:=-0.3; mv[1,1]:= 0.1; mv[1,2]:= 0.28;
mv[2,0]:=-0.4; mv[2,1]:=-0.2; mv[2,2]:=-0.75;
mv[3,0]:=0.44; mv[3,1]:=0.25; mv[3,2]:= 1.00;
translator.Add('M', mv);
end;
```
The code above adds to the translator instance the ‘µ’ variable with value 0.0, the array variable ‘A’ with 4 components, the array variable ‘B’ with 10 components and the 4x3 matrix ‘M’.

The first problem to solve is calculation of statistics square deviation from the mean value for the ‘B’ array. Math formula for the deviation of discrete random variable is (https://en.wikipedia.org/wiki/Standard_deviation):

where µ is the mean value

N is the number of elements.

This formula can be evaluated with using array operations by the following code:
```var
m, d: TValue;
begin
m:= translator.Calculate('∑B/#B');
translator.Variables['µ'].Value:= m;
d:= translator.Calculate('∑((B-µ)^2)/#B');
// print the ‘d’ value
end;```
First we calculate the mean value. The number operator ‘#’ in the formula returns the array element count. Then we assign the calculated value to the ‘µ’ variable and, finally, calculate the deviation value. Notice that there are the parentheses for the square operation in the formula, because the summation operation has higher priority. Here is the output for the evaluation:

µ = ∑B/#B = 0.06
D = ∑((B-µ)^2)/#B = 0.02876

Next problem is solving an over-determined linear equation system with the least squares approach (https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)). Let the system is set up by the ‘M’ matrix and the ‘A’ vector. Then the solution ‘X’ is defined by the formula:

where MT – is the transposed matrix, 1 means the inversed matrix.

This formula can be evaluated with the Analytics library as the following ‘M'*A/(M'*M)’. Here the apostrophe ‘'’ operator is for transposition operation and the division is equivalent to the multiplication by the inversed matrix. Evaluating the formula we get the following output:

X = M'*A/(M'*M) = TArray<TFloat>[3]=
(0.459953357586789 0.163947437790482 0.113927221302585 )

Here the ‘X’ array is the solution vector that minimizes the square error for the over-determined system.

As can be seen from the examples above, the realized array/matrix operations allow easily solve complicated problems with small, compact formulae. The complicated code for the array/matrix operations implemented inside the Analytics library and totally integrated with analytical evaluations. For an example, linear algebra operations allow solving so called ‘matrix’ ordinary differential equations (https://en.wikipedia.org/wiki/Matrix_differential_equation).

The TMS Analytics and Physics pack version 2.2 can be downloaded here. Total source code of the example application can be downloaded here .

Nancy Lescouhier

Bookmarks:

#### This blog post has received 1 comment.Add a comment.

Previous  |  Next  |  Index