New linear algebra operations and symbolic features in TMS Analytics & Physics library

Bookmarks: 

Friday, July 13, 2018

TMS Analytics & Physics developing library supports operations with array and matrix variables and includes special package of linear algebra functions. New version 2.5 introduces many capabilities and improvements for linear algebra operations. First, new operators introduces specially for vector and matrix operations: cross product operator ‘×’, matrix inverse operator ‘`’, norm operator ‘||’. In previous version of Analytics library matrix multiplication with standard linear algebra rules were implemented using multiply operator ‘*’. In new version matrix-matrix and matrix-vector multiplication implemented using the cross product operator, and the multiply operator used for by-element multiplication. This leads to more compliant algebra rules: multiply operation is always commutative, as for numbers as for matrices and vectors, and the cross operation is not commutative. By the same reason, the inverse operation used now for calculating inverse of square matrices, as the division operation now is always by-element one (so, the division operation is the inverse of multiplication). The norm operator evaluates the L2 norm of vectors and matrices, and the absolute operator ‘||’ used for evaluating absolute values of vector and matrix components. Let us consider the difference between multiplication operator and cross operator with the simple example. Suppose there are the following two matrix variables:
A [3x3] =       		  	B [3x3]=   
( 0 0.1 0.2)				( 1 0 0)
( 1 1.1 1.2)      			( 0 2 0)
( 2 2.2 2.3)      			( 0 0 3)
Evaluating expressions for multiplying the matrices using cross and multiply operators we get the following results:
A×B	= [3x3] =            	A*B	= [3x3]= 
( 0 0.2 0.6)				( 0   0   0)
( 1 2.2 3.6)				( 0 2.2   0)
( 2 4.4 6.9)				( 0   0 6.9)
As can be seen from evaluation result, the first operation implemented with standard linear algebra rules for matrix product (https://en.wikipedia.org/wiki/Matrix_multiplication), as the second operation is just by-element multiplication of the matrices. The latter operation frequently used in signal and image processing algorithms (https://en.wikipedia.org/wiki/Hadamard_product_(matrices)). Combination of both operations can be used in one expression.

In addition to the operators, new functions for matrix arguments have been introduced in Linear Algebra package: trace of a square matrix tr(M), determinant of a square matrix det(M), adjoint of a square matrix adj(M), condition number of a square matrix cond(M), pseudo-inverse of a rectangular matrix pinv(M). The second improvement concerns symbolic capabilities for vectors and matrices. In previous versions, the only way of using arrays in formulae was introducing variables for them. The version 2.5 allows using explicit vector and matrix expressions in any symbolic formula. Example of simple vector expression with four components ‘[sin(x) cos(y) e^-x a*b]’. Matrix expressions are 2D rectangular arrays of expression elements. For example, ‘[[i j k] [sin(a) cos(a) -1]]’ is a 2x3 matrix expression.

Vector and matrix expressions can be used in formulae like vector and matrix variables. The difference is that the former are unnamed but their components can depend on other variables. Thus, we can calculate symbolic derivatives of the expressions. For example, the derivative of the above vector expression by ‘x’ is ‘[cos(x) 0 –(e^-x) 0]’.

For evaluating values of vector and matrix expressions, all their components must have the same type. The component types supported are: Boolean, Real and Complex.

As example of evaluating vector expressions let us consider the following problem: there is the array ‘A’ with four components and we need to calculate new array, selecting components from the ‘A’ by different four conditions. For solving this problem we can use vector form of ‘if’ function. The expression can be the following: ‘if{[a>1 b<3 a-x>0 b+x<2]}(A [-1 -2 -3 -4])’. Here the condition of the ‘if’ function defined by the vector expression with four Boolean expression components, depending on other variables: ‘a>1’, ‘b<3’, ‘a-x>0’ and ‘b+x<2’. The arguments of the functions are array variable ‘A’ and vector expression with four real components. Evaluating the expression with some given variable values, we get the following result:
x = 1.5
a = 1
b = -2
A = (1 0.5 0.25 0.125 )

if{[a>1 b<3 a-x>0 b+x<2]}(A [-1 -2 -3 -4]) = (-1 0.5 -3 0.125 )
The example demonstrated that vector and matrix expressions can be used together with array and matrix variables and, in some cases, they are more flexible, because their components can depend on other variables.

These new improvements of linear algebra capabilities allowed realizing vector basis for linear least squares approximation. Here is the source code of an approximation example using the vector basis:
var
  approximator: TLinearApproximator;
  basis: TLinearBasis;
  func: string;
  parameters: TArray<TVariable>;
  exponents, cValues: TArray<TFloat>;
  A: TVariable;
  dim, order: integer;
begin
  // Setting parameters for approximation.
  dim:= 1; // One dimension.
  // exponent values
  exponents:= TArray<TFloat&ht;.Create(0.0, 0.5, -0.5, 0.7, -0.7); 
  order:= Length(exponents); // Basis order is the size of the array.
  // Vector variable for exponent array.
  A:= TRealArrayVariable.Create('A', exponents); 
  parameters:= TArray<TVariable>.Create(A); // Parameters for Vector basis.
  func:= 'e^(A*X[0])'; // Vector function for basis - returns real vector.
  
  // Create the basis with the specified function and parameters.
  basis:= TLinearVectorBasis.Create('X', 'C', order, dim, parameters, func);

  approximator:= TLinearLeastSquares.Create; // Linear approximator.
  // Calculate the optimal basis coefficients.
  cValues:= approximator.Approximate(basis, vData, fData); 

  // Using the approximation (basis with optimal coefficients)...
end;	
As can be seen from the code, approximation basis function has very compact form – ‘e^(A*X[0])’, because it uses vector variable ‘A’ and so, the return value of the function is real vector. The result of the approximation for some given data is the following:

F(X, C, A) = C•e^(A*X[0])

F(X) = [0.015894261869996 -0.000559709975637631 0.560703382127893
             6.67799570997929E-5 -0.60704298180144]•e^([0 0.5 -0.5 0.7 -0.7]*X[0])


Advantages of using vector basis are:
  • Simple basis function expression (in vector form).
  • Easily create high-order basis (using big array parameter).
  • Use arbitrary coefficients for the basis (fractional exponents, powers and so on).
  • Change the basis without changing the function (use other values of the parameter array ‘A’).
In addition to the vector and matrix expressions, there are some other symbolic features in the version 2.5. One feature is calculating symbolic expression for n-order or mixed derivative with one function call. The code below calculates mixed derivative of a complicated expression:
var
  translator: TTranslator;
  F, Dxy: string;
begin
  translator:= TTranslator.Create;

  F:= '(y-A*ln(y))/x+B*(v(x+y))^(2/Pi)';
  Dxy:= translator.Derivative(F, TArray<string>.Create('x','y'));

  // using mixed derivative Dxy...
end;
The output result for the derivative is:
F(x,y)  = (y-A*ln(y))/x+B*(v(x+y))^(2/Pi)
dF/dxdy = (-1+A/y)/x^2+B*(1/p-1)*(x+y)^(1/p-2)/p
Note, that after derivative evaluation the result expression simplified. New simplifications in version 2.5 are: put minus operator under sum and product expressions (first expression in the sum ‘(-1+A/y)/x^2’); simplification of square root and power expressions (second expression in the sum ‘B*(1/p-1)*(x+y)^(1/p-2)/p’). Finally, there is new symbolic function that allows getting all variable names from math expressions. Here is an example of using the function:
var
  F: string;
  Names: TArray<string>;
begin
  F:= 'A[i]*sin(2*Pi*x)+A[j]*cos(2*Pi*y)+B[i]*e^((1+I)*x)+B[j]*e^((1-2I)*y)';
  Names:= TTranslator.VariableNames(F);

  // using the Names array...
end;
The result is the following:
F = A[i]*sin(2*Pi*x)+A[j]*cos(2*Pi*y)+B[i]*e^((1+I)*x)+B[j]*e^((1-2I)*y)
Variable names: A, i, x, j, y, B
Note, that the function does not include variable names in the result twice (i, j, x, y, A, B). In addition, standard constants, like Pi and Euler number, not included in the names list.

The version 2.5 is already available here. Source code of the example application can be downloaded from here. Other examples of using vector/matrix operations, expressions and vector basis with graphics visualization included in demo projects of the version 2.5.

Bruno Fierens


Bookmarks: 

This blog post has not received any comments yet. Add a comment.



TMS RADical WEB, preferred error handling?

Bookmarks: 

Thursday, June 28, 2018

With almost all in the TMS development team and since some time also many TMS ALL-ACCESS users busy using TMS WEB Core to develop web client applications from the Delphi IDE, we wonder about what could be the preferred developer experience to deal with run-time errors?

To situate, there are different categories of errors, but to focus we'll divide the types of errors in two categories:

  • Delphi code causing exceptions
  • DOM manipulations causing errors
A Delphi code exception is for example an out of bounds access in an array. A DOM error is for example accessing a non-existing attribute of a HTML DOM element or a network error. The standard behavior in the browser is that the error is displayed in the console. To see the error, the developer needs to open the console (for example with the F12 in the Google Chrome browser). When not looking in the console, errors go in many cases unnoticed.

As a software developer, we're interested in all sorts of errors so we can locate the cause and apply fixes to avoid the errors. From the feedback of several users already busy with TMS WEB Core, we learned that in many cases an unexpected behavior is reported but it is overlooked to look in the console to get more details about what exactly might have happened. This made us reflect on how we can do better to make it clear to developers that an error happened in the web application. There are several possibilities and perhaps there are even more we didn't think about yet.

For an error, like an out of bounds exception that looks in the console like:



some alternative possibilities are:

1) Show the error as an alert (modal dialog in the browser):



Advantage is that it is blocking and thus can't be ignored but the position of the alert is controlled by the browser and somewhat awkward. Disadvantage is that we need to click the dialog away for every error that happens.

2) Show the error with a nicer TMS WEB Core dialog that makes it more clear it concerns an error:



Advantage is that it is centered in the window and cannot be misinterpreted that it concerns an error. Disadvantage here is also that we need to click the dialog away for every error that happens.

3) Show the error in a red colored area at the bottom of the browser window:



Advantage is that it is less intrusive and the red colored area can host multiple errors simultaneously when it grows.

4) Let the developer decide what to do from the Application object OnError event:

procedure TForm1.AppErrorHandler(Sender: TObject; AError: TAppplicationError;
  var Handled: boolean);
begin
  ShowMessage(AError.AMessage+#13+AError.AFile+#13+inttostr(AError.ALineNumber)+':'+inttostr(AError.AColNumber)+#13+AError.AStack);
end;

5) Show the error in the Delphi IDE:



This is far from trivial but we have some experimental code & proof of concepts that could make this perhaps in the future also a feasible solution.

6) Other solutions?:

Perhaps there are other possibilities we didn't think about yet, so we're curious to hear your thoughts!

We had some internal discussions with the development team here and there are different opinions. So, therefore, we wonder what your thoughts are and what you think is the preferred way to handle errors for your Delphi web client applications. Note that the default behavior could even be different in DEBUG mode from RELEASE mode.

You can vote here what your preferred error handling method is or you can leave comments on this blog:


Bruno Fierens


Bookmarks: 

This blog post has received 5 comments. Add a comment.



Object Pascal: Compiler Directives

Bookmarks: 

Tuesday, June 26, 2018

Photo by Mathyas Kurmann on Unsplash

Compilation Directives could help you to make your code multi-platform or even cross-compiled.

Introduction

Compilation directives are powerful commands that developers could use to customize how the compiler works.

These directives pass parameters to the compiler, stating the arguments of the compilation, how must be compiled, and which will be compiled.

There are basically 3 types of compilation directives.

  • Switch directive
  • Parameter directive
  • Conditional compilation directive

The first two types change the compile parameters, while the last one changes what the compiler will perform on.

In this article we will deal with the last one: Conditionals.

Conditionals

They are powerful commands.

With just a few conditional commands, your Object Pascal code can be compilable across multiple platforms.

However, as we add more and more directives, the code will become more complex.

Let's take a look in the example below:

 //http://docwiki.embarcadero.com/RADStudio/Tokyo/en/Conditional_compilation_(Delphi)
  {$DEFINE DEBUG}
  {$IFDEF DEBUG}
  Writeln('Debug is on.'); // This code executes.
  {$ELSE}
  Writeln('Debug is off.'); // This code does not execute.
  {$ENDIF}
  {$UNDEF DEBUG}
  {$IFNDEF DEBUG}
  Writeln('Debug is off.'); // This code executes.
  {$ENDIF}

In this example, only the first and third Writeln function calls will be executed.

All directives and also the second function call won't be part of the final executable, I mean, the ASSEMBLY code.

Cool.

However, it looks like that the code is "dirty" and also we have a temporal coupling, because the constants need to be defined in a specific order.

Directives and definitions of constants that will be used in only a single unit may even be manageable, but what if you will work with tens or even hundreds of units that will use these directives and definitions, do you still think this approach is the best choice for the architecture of your project, with the purpose of building it as cross-compiled or multi-platform?

I don't think so.

Encapsulating Directives

Imagine a project that needs to be cross-compiled in Delphi and Free Pascal. We would like to use the same classes, the same code, but could exists some differences between these compilers.

The code needs to evolve independent of the compiler. I mean, if some changes could be done to improve the code when it is compiled on Free Pascal, for example, it should be done without thinking in some difference that might exist in Delphi.

To do so properly, we could not work in a code with many compilation directives, because some changes could broke the Delphi version or vice-versa.

Instead of seeing directives, it might be better seeing just classes.

I would called this, encapsulated directives.

Implementation

Imagine an unit witch contains a class to represents a MD5 algorithm.

Free Pascal already has a md5 unit which has functions to do the job and of course we have to make classes to encapsulate those functions.

In Delphi, the unit that do the same job is named hash.

We do not want to "reinvent the wheel" then, let's use what is already done on both platforms.

So, how would you do this implementation neither using conditional directives in implementation code nor *.inc files?

First of all, let's create our MD5 unit:

unit MD5.Classes;

interface

uses
  {$ifdef FPC} MD5.FPC {$else} MD5.Delphi {$endif};

type
  TMD5Encoder = class(TEncoder);

implementation

end.

As you can see, there is almost nothing on this unit. The real code will stay in others units, i.e, MD5.FPC and MD5.Delphi, one for each platform.

Let's create the FPC version:

unit MD5.FPC;

interface

uses
  md5;

type
  TEncoder = class
  public
    function Encode(const Value: string): string;
  end;

implementation

function TEncoder.Encode(const Value: string): string;
begin
  Result := MD5Print(MD5String(Value));
end;

end.
Those functions with prefix "MD5" comes from md5 unit.

Then, let's create the Delphi version:

unit MD5.Delphi;

interface

uses
  hash;

type
  TEncoder = class
  public
    function Encode(const Value: string): string;
  end;

implementation

function TEncoder.Encode(const Value: string): string;
begin
  Result :=THashMD5.GetHashString(Value);
end;

end.

Both units have the TEncoder class definition (yes, same name in both). Then, we created a TMD5Encoder class that inherits from the correct class, which is platform dependent, and voila! We have clean units, without any conditional directives inside methods.

Finally, we have two distinct classes in different units that can evolve its implementation independently, without any fear of breaking the code between platforms.

Conclusion

Compilation directives is a good tool for customize the code. However, it should be used with parsimony.

In object-oriented code, try to use more specialized objects than compilation directives.

For each conditional directive that you want to add to the code, I suggest implementing a new class that encapsulates the directive or a set of them.

Your code will be cleaner and sustainable.

See you.



Marcos Douglas B. Santos


Bookmarks: 

This blog post has not received any comments yet. Add a comment.



TMS Day followup: Using TMS FlexCel with TMS WEB Core

Bookmarks: 

Wednesday, June 20, 2018

Another TMS day

They say time flies when you are having fun... It is hard to believe it has been about 2 years since my last blog post, which was also about a TMS Day. I see two options here: Either we start doing more TMS Days so I can blog about the sessions, or I just start actually writing down more stuff.

On my discharge I'd like to say that I haven't been completely silent this 2 years: I've been writing the stuff I would normally write in a blog post in the "Tips and Tricks" on FlexCel docs. Ok, this is a poor excuse, but an excuse anyway and I have nothing better so I'll stand by it.

Now, it is time to go back to the TMS Day. As always, it was a great experience to be there and speak face to face with customers and sharing experiences . Me sitting here and writing a blog post just isn't the same. And to be brutally honest, speaking with the people did change my mind, and this is not just something I am saying because it sounds nice. When I arrived to the TMS Day I saw WEB Core as an interesting technology, but I saw little connection with it and the FlexCel reporting stuff, which is what I do. Reporting is normally a server-side thing, to be done on the server where the database is, and not something you would do normally client-side in javascript. So, there is little to integrate: you can call FlexCel from WEB Core, but you can call any other reporting solution too. You can also use WEB Core with FlexCel, but you can use any other web solution with FlexCel too. When I arrived, this use case (WEB Core in the client, reports with FlexCel in the server) was the only one I had in mind. When I was there, I spoke with a lot of people and learned about a lot of new use cases I hadn't even thought about. When I left, I had a lot of more plans for a deeper integration. But more on that below.

FlexCel as a server side solution for reporting

We started the session with a very basic description of what FlexCel does. As usual, this was a very packed session and I had only half an hour, so I didn't wanted to waste a lot of time explaining what FlexCel did. We just covered the basics: With FlexCel you create Excel files either with code or by writing tags in an Excel template, and then you can export those files to pdf or html. While we use Excel as a building block for the report, it is not needed to have the result in Excel. And as this session was all about the web, we just created pdf and html reports.

1. A simple report

Some time ago I read about a really interesting project on where they used a raspberry pi to record how many times a day a baby would say the word "why". Being a father of a little 4 year old myself, I was very interested in the experiment and I thought it could be an interesting real world example on something that needed reporting. At the same time, with me also being lazy and with more stuff to do than I could possibly finish in five lives, I decided that instead of actually recording my child, I would just create a small database with my own estimates on how many times she would pronounce the word.

I ended up with an access database that looked like this:



Yes, it was in Access, and you wouldn't use Access for a real application. Bit for a demo, it is fine to me. Just remember to use a real database in real applications.

The next step was creating two different applications. Since TMS WebCore runs on the client (javascript) and FlexCel is a server side product which runs in the server (delphi), I needed to create both a TMS webcore app for the client, and a Delphi WebBroker app for the server.

1.1. Server side

Server side we created a new WebBroker app by going to File->New->Other... then choosing "WebBroker" in the left panel and "Web Server Application" in the right.

The focus on this session wasn't in how to create a FlexCel report or a WebBroker app, so I just used a pre-made example which you can download at the end of this post.

For the Excel report I just created an Excel file that looked like this:



I just wrote tags with the database fields (like <#data.measureddate>) in cells A3 and B3, then defined a name where the report would run and added a chart as to make it a little more interesting. Note: Right now FlexCel renders only charts in xls files to pdf and html, and while we are working in fully supporting xlsx this is not yet ready. So this is why I used an xls file instead of an xlsx.

I also added some conditional formatting just for fun:



After all, one of the nicest parts of doing reports in FlexCel is that you get access to all those simple features in Excel like conditional formats, tables or charts, so why not to use them? Even if in a case like this, I am not generating Excel files at all, only pdfs and html files. That was all I did in the template. Then in the Webbroker app, I added a datamodule to access the database, and the following code to the DefaultHandlerAction event:
procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
var
  Report: TFlexCelReport;
  Xls: TExcelFile;
  Ts: TMemoryStream;
  Pdf: TFlexCelPdfExport;
begin
  Report := TFlexCelReport.Create(true);
  try
    Report.AddTable('data', datamodule1.data);

    Xls := TXlsFile.Create('....why.xls', true);
    try
      Ts := TMemoryStream.Create();
      try
        Report.Run(Xls);
        Pdf := TFlexCelPdfExport.Create(Xls, true);
        try
          Pdf.BeginExport(Ts);
          Pdf.ExportSheet;
          Pdf.EndExport;
        finally
          Pdf.Free;
        end;

        Response.SendStream(Ts);
      finally
        Ts.Free;
      end;
    finally
      Xls.Free;
    end;
  finally
    Report.Free;
  end;

This basically adds the datasource "datamodule1.data" to the report, then runs the report and then exports it to pdf, sending the pdf bytes to the Response of the html app. If you now run the application, and press the "Open Browser" app in the Webbroker form, you should see something like this:



Note that until here, we haven't used TMS WebCore at all. This is just a server side app which runs a report over a database, and returns a pdf when you call it. Note also how we got the conditional formatting we wrote in the template for column B shows in the final report, and the chart is filled with data.

1.2. Client Side

Ok, now what about the client side? Imagine we have a TMS WebCore app, and we want to show the report when an user clicks a button on it. How do we call the server app we coded just moments ago?

To answer the question, we will create a new TMS WebCore application, and drop a WebButton:



Then, we double click the button, and write the following code:
uses ... Web,...;
...
procedure TForm1.WebButton1Click(Sender: TObject);
begin
    window.open('http://localhost:8080');
end;
And that should be it! Now when you run the tms WebCore application and press the button, it should open the report. Note that of course the server must be running and listening in localhost:8080 for this to work.

2. A more complex integration

In the previous section we saw a very simple example on how to integrate reporting in WebCore. There was basically no integration between WebCore and FlexCel, and all the above can be resumed in the lines:
uses ... Web,...;
...
procedure TForm1.WebButton1Click(Sender: TObject);
begin
    window.open('http://localhost:8080');
end;
You could have used any other reporting solution here instead of FlexCel, and just have a webserver app exposing the reports as html or pdf. It doesn't get any simpler than this, and I believe this is a good thing. Calling window.open will probably be the most used way to show reports from a WebCore app.

But what if we wanted a little more integration? Well, you can do that too, and this is what we covered in the second demo. As for the first demo we had used PDF, on this second demo we will be showing HTML reporting. And while we could call the html reports also with window.Open as in the first case, here we want to integrate the report inside the TMS WebCore app.

2.1. Server Side

Server side, we are going to use the same template that we used in the first example, but export to HTML instead of PDF. We will use a FlexCelHTMLExport class instead of a FlexCelPDFExport class.

We are also going to embed the images in the HTML file (using HTML5) so we don't have to feed separated images and html to the html component that is going to display the report client side. We are also going ot set the image resolution to 192 dpi so the chart looks crisp in the generated file.
procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
var
  Report: TFlexCelReport;
  Xls: TExcelFile;
  Ts: TMemoryStream;
  Writer: TStreamWriter;
  Html: TFlexCelHtmlExport;
begin
  Report := TFlexCelReport.Create(true);
  try
    Report.AddTable('data', datamodule1.data);

    Xls := TXlsFile.Create('......why.xls', true);
    try
      Ts := TMemoryStream.Create();
      try
        Writer := TStreamWriter.Create(Ts, TEncoding.UTF8);
        try
          Report.Run(Xls);
          Html := TFlexCelHtmlExport.Create(Xls, true);
          try
            Html.HtmlVersion := THtmlVersion.Html_5;
            Html.EmbedImages := true;
            Html.ImageResolution := 192;
            Html.Export(Writer, 'report.html', nil);
          finally
            Html.Free;
          end;

          Response.ContentStream := Ts;
        finally
          Writer.Free;
        end;
      finally
      //don't free the content stream
       // Ts.Free;
      end;
    finally
      Xls.Free;
    end;
  finally
    Report.Free;
  end;

2.2. Client Side

We are going to create a new TMS WebCore application, drop a button, a WebHTMLContainer, and a WebHTTPRequest component:



This time we are going to use the WebHTTPRequest component to get the report. WebHTTPRequest is async, so we need to call it from the button click event, and then when the answer data is available, load it into the WebHTMLContainer. The first step then is to set the server address in the WebHTTPRequest properties:



Note that for this example we will use port 8081 instead of 8080 as in our last example, so we can run both at the same time. This means that we also need to change our server app to serve in port 8081 instead of 8080.

Then, on the button click event let's call the WebHTTPRequest, and let's also write the Response event of the WebHTTPRequest so it fills the WebHTMLContainer with the data:
procedure TForm1.WebButton1Click(Sender: TObject);
begin
  WebHttpRequest1.Execute;
  WebButton1.Enabled := false;
end;

procedure TForm1.WebHttpRequest1Response(Sender: TObject; AResponse: string);
begin
  WebButton1.Enabled := true;
  ShowMessage('Ok!');
  WebHTMLContainer1.HTML.Text := AResponse;
end;

A last note. At the time of this writing, TWebHTMLContainer doesn't have a property to change how it handles overflow. But as TMS WebCore allows full control on the generated html and css, we are going to apply a little hack to make the WebHTMLContainer scrollable. We will edit the Project2.html file, and add the following to the head section:
    <style type="text/css">
    #TForm1_HTMLContainer1 {overflow: visible!important}
    </style>
This line will allow the container to scroll, and as it is defined as !important, it should overwrite other rules that make the container not to scroll. This should be just a temporary hack, until the WebHTMLContainer gets a property that allows you to manipulate this directly.

So after all of this, it is time to run the application! And we get...



CORS

What was that error? Everything was going so well! But let's not panic: The problem here is with something called Cross-Origin Resource Sharing (CORS) Basically, we are trying to access a resource in the server (localhost 8081) from the client (localhost 8000) and this is not allowed by default. We need to specifically allow requests from localhost 8000 into the server.

And you allow the requests by setting the headers Access-Control-Allow-Origin and Access-Control-Allow-Headers

The full code in the server should now be:
procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
var
  Report: TFlexCelReport;
  Xls: TExcelFile;
  Ts: TMemoryStream;
  Writer: TStreamWriter;
  Html: TFlexCelHtmlExport;
  HtmlState: TPartialExportState;
begin  
  Response.SetCustomHeader('Access-Control-Allow-Origin', '*');
  Response.SetCustomHeader('Access-Control-Allow-Headers', 'X-Custom-Header, Cache-Control');

  Report := TFlexCelReport.Create(true);
  try
    Report.AddTable('data', datamodule1.data);

    Xls := TXlsFile.Create('......why.xls', true);
    try
      Ts := TMemoryStream.Create();
      try
        Writer := TStreamWriter.Create(Ts, TEncoding.UTF8);
        try
          Report.Run(Xls);
          Html := TFlexCelHtmlExport.Create(Xls, true);
          try
            Html.HtmlVersion := THtmlVersion.Html_5;
            Html.EmbedImages := true;
            Html.ImageResolution := 192;
            HtmlState := TPartialExportState.Create(TFlexCelWriter.Null, '');
            try
              Html.Export(Writer, 'report.html', nil);
            finally
              Html.Free;
          end;

          Response.ContentStream := Ts;
        finally
          Writer.Free;
        end;
      finally
      //don't free the content stream
       // Ts.Free;
      end;
    finally
      Xls.Free;
    end;
  finally
    Report.Free;
  end;

end;

With this fixed, we should now see the application working with the report embedded inside it:



3. Even more integration

We didn't got to cover this section in the TMS day, as it was already too much to say for 30 minutes, but I would like to mention it here anyway. The last example (the "integrated" one) does work, but it is not really ok. It is embedding a full HTML document (the one created by FlexCel) inside another HTML document (the TMS WebCore app), and that is not valid HTML. We could have used an iframe instead, but iframes are so ridden with security issues that it might be a solution worse than the problem.

To do a nice integration, we need to separate the HTML headers from the body. Luckily FlexCel HTML exporting was designed from the start to allow you to get the different parts of the html document so you can integrate the HTML output in your site.

3.1. Server Side

On the FlexCel side, we need to use a TPartialExportState object instead of writing directly to an html file. TPartialExportState allows us to get the different parts of a report instead of the full text.

The next thing is that while we want to separate the css from the html so we can write them in different places in the client, we would like to keep everything inside a single request to the server. So we will be sending an encoded response, where the first part is the css, then we have a 4-char 0 separator, and then we have the html. Client side, we will separate this response back to the 2 original parts, and write them in the correct places.

So server side, the code ends up as follows:
procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
var
  Report: TFlexCelReport;
  Xls: TExcelFile;
  Ts: TMemoryStream;
  Writer: TStreamWriter;
  Html: TFlexCelHtmlExport;
  HtmlState: TPartialExportState;
begin
  Response.SetCustomHeader('Access-Control-Allow-Origin', '*');
  Response.SetCustomHeader('Access-Control-Allow-Headers', 'X-Custom-Header, Cache-Control');

  Report := TFlexCelReport.Create(true);
  try
    Report.AddTable('data', datamodule1.data);

    Xls := TXlsFile.Create('......why.xls', true);
    try
      Ts := TMemoryStream.Create();
      try
        Writer := TStreamWriter.Create(Ts, TEncoding.UTF8);
        try
          Report.Run(Xls);
          Html := TFlexCelHtmlExport.Create(Xls, true);
          try
            Html.HtmlVersion := THtmlVersion.Html_5;
            Html.EmbedImages := true;
            Html.ImageResolution := 192;
            HtmlState := TPartialExportState.Create(TFlexCelWriter.Null, '');
            try
              Html.PartialExportAdd(HtmlState, 'report.html', '', true);
              //We will encode the css and body in the same Response.
              HtmlState.SaveCss(Writer);
              Writer.Write(#0#0#0#0); //Use 4 character 0 as separator between CSS and Body.
              HtmlState.SaveBody(Writer, 1, '');
            finally
              HtmlState.Free;
            end;
          finally
            Html.Free;
          end;

          Response.ContentStream := Ts;
        finally
          Writer.Free;
        end;
      finally
      //don't free the content stream
       // Ts.Free;
      end;
    finally
      Xls.Free;
    end;
  finally
    Report.Free;
  end;

end;
This code will output the css , then the separator #0#0#0#0, then the html.

3.2. Client Side

Client side, we need to re-separate the css and the html, then output the html into the Text of the WebHTMLContainer, and the css into the header of the page.

To output the html, we will use the same code as in the last example. To output the css, we will use the method AddInstanceStyle in TControl. Since this method exists only in web controls but not in normal Win32 controls, we need to write the method inside {$IFNDEF Win32} defines.

There is one last issue to address: FlexCel returns the full css including the enclosing <style> and </style> tags. But AddInstanceStyle expects the inner HTML without the <style> tags. So for this example, we just had to do some poor man's parsing and manually remove the <style> and </style> tags in the css returned by FlexCel. This should not be necessary in the future, since for FlexCel 6.20 we are adding a new overload TPartialExportState.SaveCss which has a parameter includeStyleDefinition. Once we release 6.20, you will be able to just call SaveCss with includeStyleDefinition = false, and there will not be a need to remove it manually to call AddInstanceStyle.

So finally, the code in the client ends up like this:
procedure TForm1.WebHttpRequest1Response(Sender: TObject; AResponse: string);
var
 cssPos: integer;
 cssString: string;
 innerCssStart, innerCssEnd: integer;
begin
  WebButton1.Enabled := true;
  cssPos := Pos(#0#0#0#0, AResponse);
  if (cssPos < 1) then
  begin
    ShowMessage('Invalid response from server');
  end
  else
  begin
      ShowMessage('Ok!');
  end;

  WebHTMLContainer1.HTML.Text := copy(AResponse, cssPos + 4, Length(AResponse));
  cssString := copy(AResponse, 1, cssPos - 1);
  innerCssStart := pos('>', cssString) + 1;
  innerCssEnd := pos('', cssString);
  {$IFDEF WIN32}
  ShowMessage(copy(cssString, innerCssStart, innerCssEnd - innerCssStart));
  {$ELSE}
  WebHTMLContainer1.AddInstanceStyle(copy(cssString, innerCssStart, innerCssEnd - innerCssStart));
  {$ENDIF}
end;

With this code, the document should be a valid html document, with the css in the header and the html for the report inside the div. The results will look similar as in example 2, but the html is now correct.

4. Hyperlinks

Ok, why stop now? There is still so much to cover! But well, I only had 30 minutes to speak so in the TMS day I stopped in the second demo. In this post we have a little more time, so I'll cover one big pink elephant in the room I managed to ignore up to now: Hyperlinks.

HTML is supposed to be about hyperlinks, it is right there on the first "H" in HTML. Wouldn't it be cool if we could add hyperlinks to our reports? I will answer this for you: indeed it would be cool. We could do for example a drill down report where we list the whys per day as in the previous examples, but when you click on a date, you can see a detail on how the whys evolved during that day. And this is exactly what we are going to try to do in this part.

4.1. Server Side

Server side, we will have to modify our template, and add a new one for the details. We will rename why.xls to why-master.xls, and add an hyperlink at cell A3. The hyperlink will have as target the URL:

http://tmsexample.com/detail?day=*.data.measureddate.*



As in Excel you can't write FlexCel tags like <#data.measureddate> inside an hyperlink, FlexCel allows the alternative syntax *.data.measureddate.* instead, and that is what we used above.
We also used a bogus domain name, tmsexample.com because Excel only understands absolute hyperlinks. When exporting to html with FlexCel, we will set the BaseUrl property in the FlexCelReport component to:
html.BaseUrl := 'http://tmsexample.com';
and that will make the url relative by removing the start of it. We could have used any other domain name here, the only thing required is that we use the same name in the template and in the BaseUrl property.

We will also create a why-template.xls which is similar to the master, but with different formatting and a link to go back in cell A1 instead of a link to drill down in cell A3.



Next step would be to add a new DataSet to the datamodule to do the query for one day, instead of the query grouped by days that we had before.

And finally, the server now needs to handle requests for the detail and for the master. We will do this by adding a new action to the webbroker module:



The actions now are like this:
procedure TWebModule1.WebModule1DefaultHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
begin
  RunReport('why-master.xls', datamodule1.data, TDateTime(0));
end;

procedure TWebModule1.WebModule1DetailHandlerAction(Sender: TObject;
  Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
var
  date: TDateTime;
  day: string;
begin
  day:= Request.QueryFields.Values['day'];
  date := Trunc(StrToDate(day));
  DataModule1.detail.Parameters.ParamByName('day').Value := date;

  RunReport('why-detail.xls', datamodule1.detail, date);
end;
Where "RunReport" is a generic method that is more or less similar to the previous examples. We are not going to show it here since it is not interesting, and as with the other examples, the full source code is at the end of this post anyway.

Note: In this particular case I am using the dates without any preprocessing as parameters for the detail report, but this is far from ideal. You can never know if 1/3/1999 is March 1 or January 3, and client and server might understand dates differently, so the best solution is to use a neutral format like "yyyy-mm-dd" or even the date serial number. But to keep this example simple, I just used the localized dates as parameters.

With these modifications, if you now run the server app, you should see a screen like this:



And if you now click on a date, say January 8, you should see the details for that day:



You can click in the arrow at the top to go back to the main report.

4.2 Client Side

Finally, the last thing to study is how to make this work as a TMS WebCore application. At first sight, we might think that the application from the last example should work: after all the master report has relative links to the details, and the details to the master, so it all should work transparently. It is already working if you call the webpages directly.

But, if you try the application, you will notice that the master report loads fine, but when you click on the links, you get an error:



The explanation on why this is happening is simple, but it might not be completely intuitive. If you look at the Url in the previous screenshot, you'll see it points to http://localhost:8000/detail?day=1/6/2018. But the server is running in port 8083, so that is why the error. The relative Url in the report is being resolved with the address of the client, not the server.

The approach here works for a separate report (one that you would open with Window.Open) but it just won't work for reports that are integrated in the app. For that, we need to dig a little bit further.

5. Hyperlinks in integrated reports

As we have seen in point 4, when running integrated reports, we can't just have the reports link to other subreports. Those links would take us away from our app, which is running as a single page in the client.

What we need to do is to convert those links in the master report to javascript calls, and then modify our app to handle those javascript links and open the correct report inside our application.

5.1. Server side

Server side, we now need to generate Javascript links, not links to a different page.While we are at it, we will use the date serial number instead of the date string as the parameter, as to avoid the problem of what date is "1/2/1999" which we mentioned in the previous section.

As the expression in the hyperlink is going to get a little complex, we are going to start by creating a config sheet in the master report template. A config sheet is just a special sheet in a FlexCel report where you can define a lot of the stuff that goes into the report. In our case, we are going to add an expression named detaillink which we will define as:
detaillink = javascript:pas.Unit5.Form1.OpenDetail(<#evaluate(VALUE(<#data.measureddate>))>);


Then we will edit the hyperlink itself, and change it to be:
http://tmsexample.com/*.detaillink.*
Different from the previous example, now the logic on what goes in *.detaillink.* is now encapsulated in the config sheet, and we can now play with its definition without having to edit the link

The rest of the application is going to be the same as in our last example, but with a small meaningful difference. This time we are going to add a slash at the end of the BaseUrl property.
html.BaseUrl := 'http://tmsexample.com/';
That extra / at the end will make sure we remove the starting / in the url, and end up with links like href="javascript:pas.Unit5.Form1.OpenDetail(43104);" instead of href="/javascript:pas.Unit5.Form1.OpenDetail(43104);"

We will do similar modifications to the detail template, so the links call an OpenMaster() javascript method.

5.2. Client Side

Client side is where we have to do more changes. Server side, we are now generating links which call some "OpenDetail" and "OpenMaster" methods in javascript, and pass the serial number of the date as the parameter for OpenDetail. We now need to define those functions in Javascript.

Luckily TMS WebCore makes it simple. We are going to just define two new pascal methods as follows:
procedure TForm1.OpenDetail(const Date: integer);
begin
  WebHttpRequest1.URL := 'http://localhost:8084/detail?day=' + IntToStr(Date);
  WebHttpRequest1.Execute;
  WebButton1.Enabled := false;
end;

procedure TForm1.OpenMaster;
begin
  WebHttpRequest1.URL := 'http://localhost:8084';
  WebHttpRequest1.Execute;
  WebButton1.Enabled := false;
end;
And make them part of the published interface of the class.

Now, when we get a link like "javascript:pas.Unit5.Form1.OpenDetail(43104)" from the report, the browser will call the method OpenDetail in our code, and pass 43104 as a parameter. As you could see in the code above, we use that parameter to fetch the correct report from the server, and then it is loaded as any other report.

So to round it all up, and as this is a blog about a tms day, below you can see a small video showing how this small reporting app ended up working:



The future

I wasn't originally planning to speak about the future when I started preparing the presentation, as I was just planning to keep FlexCel server only. But as I mentioned at the start, after speaking with people on the TMS day I got convinced that we need to do more with FlexCel in the Javascript front end.

So where do we start? To be realistic, there is little chance that FlexCel will be compiled with TMS WebCore in the near future. FlexCel uses a lot of generics and other stuff that is not supported by the pascal to js compiler, and to be 100% sincere, we can't even compile FlexCel with Lazarus which supposedly has the features we need. (and believe me, we've tried).

But FlexCel is not only FlexCel for Delphi. We also have FlexCel for .NET which could be converted to asm.js or webassembly. Once it is javascript, it doesn't matter if we started from Pascal or C# code, it will work the same.

FlexCel .NET could be an option. But in this TMS day we introduced another possibility, which seems likely to be the one that wins. You know, there has been a third branch of FlexCel living a "secret" life since 2015. This branch is fully written in C++ 11, and C++ converts pretty well to Webassembly. I said it on tms day and I will say it again now: There is no guarantee that FlexCel for C++ will ever reach a stage where we release it. Please don't wait for it. Our priorities are in FlexCel itself, right now more specifically in FlexCel 7 which will have xlsx chart rendering and will let us do the same demos we did here with xlsx files instead of xls and still show the charts. FlexCel C++ is a side project, which gets time only when there isn't anything more urgent, which is very little time.

Now, with the disclaimer out of the way, FlexCel for C++ is actually working for small cases, and the conversion to Webassembly was so seamless that it looked like magic. So I really wanted to show it working. We took some small tests which are passing in C++, compiled them with Webassembly, and saw the results. I am not going to replicate what we did there, as the post would get too long, but I just want to share some screenshots with the final results.

This is the tests running in C++:



And here they are running in node.js:



Yes, they look the same, and no, they aren't even remotely similar. One is running compiled C++ code in a mac, the other is javascript running under node.js. And the tests aren't trivial either, they are reading and creating xlsx files already. And yes, if I change the code, I can see the assertions break in both C++ and Javascript; we did that live on tms day. It does seem kind of magical to me.

So this are the plans, or maybe the dreams right now. What the future will actually bring nobody knows, but it feels good to share those dreams with you. With a little more bit (actually a lot) of effort they could actually come true. I know I am really looking forward to have C++ and Webassembly versions of FlexCel.

Q & A

Sadly this time we didn't got time to do a Q&A, and I feel sad about it. Q&A are the parts that I enjoy more on the presentations. To make up for it, please ask your questions in the comments!

You can get the source code for all the demos and the powerpoint slides used in the presentation here: http://www.tmssoftware.biz/flexcel/samples/tmsday-flexcel-and-webcore.zip

Adrian Gallero


Bookmarks: 

This blog post has received 1 comment. Add a comment.



Article & sample for how to use Embarcadero RAD server as server backend for TMS WEB Core applications

Bookmarks: 

Wednesday, May 30, 2018

TMS WEB Core is a framework for creating web client applications from the Delphi IDE in the Delphi language and using RAD component based development. TMS WEB Core is designed to be totally open in many ways, also with respect to binding to a server back-end. This binding to the server back-end is typically done using REST APIs. TMS software has its own high-performance, flexible and low-cost REST API framework TMS XData for accessing databases on a server but it is equally possible to use node.js, ASP.NET Core microservices, other technologies or also the Embarcadero RAD Server that comes with Delphi (there is now one free license included in the Delphi Enterprise or Architect editions) and that offers a powerful, fast & flexible creation of REST API endpoint servers.

  

Embarcadero MVP, famous speaker and writer of several Delphi books Bob Swart, was excited when he learned about TMS WEB Core and wanted to explore the capabilities of using Embarcadero RAD server as a backend for a TMS WEB Core web client application. Based on the results of his research and work, Bob Swart did the effort to create an article that explains and demonstrates how one can use Embarcadero RAD server with TMS WEB Core from start to finish. There is not only the article that Bob Swart makes available but also the sample source code.

Downloads

Article:

Source code:



Get started today: Technical previews of TMS WEB Core, TMS FNC UI web-enabled controls, web-enabled TMS XData, the first parts under the TMS RADical WEB umbrella are exclusively available now for all active TMS-ALL-ACCESS customers.

Bruno Fierens


Bookmarks: 

This blog post has not received any comments yet. Add a comment.




Previous  |  Next  |  Index