Delphi and iPhone helping vision impaired people

Friday, September 30, 2016

We released earlier this week a major update of the TMS FMX Cloud Pack. This new version adds a lot of new components covering seamless access to all kinds of interesting cloud services. Among the new services covered, two services from Microsoft stand out and open up new ways to enrich our Delphi applications with cool features. In this blog, I wanted to present the Microsoft Computer Vision and Microsoft Bing speech service. Our new components TTMSFMXCloudMSComputerVision and TTMSFMXCloudMSBingSpeech offer instant and dead-easy access to these services. Powered with these components, the idea came up to create a small iPhone app that let's vision impaired people take a picture of their environment or a document and have the Microsoft services analyze the picture taken and let Microsoft Bing speech read the result.

So, roll up your sleeves and in 15 minutes you can assemble this cool iPhone app powered with Delphi 10.1 Berlin and the TMS FMX Cloud Pack!
To get started, the code is added to allow taking pictures from the iPhone. This is a snippet of code that comes right from the Delphi docs. From a button's OnClick event, the camera is started:

  if TPlatformServices.Current.SupportsPlatformService(IFMXCameraService,
    Service) then
  begin
    Params.Editable := True;
    // Specifies whether to save a picture to device Photo Library
    Params.NeedSaveToAlbum := false;
    Params.RequiredResolution := TSize.Create(640, 640);
    Params.OnDidFinishTaking := DoDidFinish;
    Service.TakePhoto(Button1, Params);
  end
When the picture is taken with the camera, the DoDidFinish() method is called, so we need to add the code there to deal with the image taken. We'll cover that in a moment. Let's first setup now the TMS FMX Cloud Pack components TTMSFMXCloudMSComputerVision and TTMSFMXCloudMSBingSpeech to consume the needed cloud services. You can download the components from the TMS FMX Cloud Pack page. After installing, over 60 new cloud service components will be added to your tool palette. It will be needed to get an API key to be allowed using these Microsoft services. Go to https://www.microsoft.com/cognitive-services/ and signup for free for the computer vision API and Bing speech API and you'll receive an API key for each. In the app, two constants are used and the component's app key property is initialized when the app starts:

procedure TForm1.FormShow(Sender: TObject);
begin
  TMSFMXCloudMSBingSpeech1.App.Key := MSBingSpeechAppkey;
  TMSFMXCLoudMSComputerVision1.App.Key := MSComputerVisionAppkey;
end;
Now that these components are also ready to be used, let's complete the DoDidFinish() method that is triggered when the picture is taken. This method returns the image taken. This image is saved locally and will be submitted to the Microsoft computer vision API for analysis. In the app, we added via a radiobutton the choice between regular image analysis or OCR.


So, a TTask is used to start this analysis with the call TMSFMXCLoudMSComputerVision1.ProcessFile(s, cv). A TTask is used to avoid that the UI is locked during this analysis, after-all, the image must be submitted to Microsoft, processed and the result returned and parsed, so this can take 1 or 2 seconds. Depending on the analysis type, the result is captured as text in a memo control. After this, we connect to the Bing speech service.

procedure TForm1.DoDidFinish(Image: TBitmap);
var
  aTask: ITask;
  s: string;
  cv: TMSComputerVisionType;
begin
  CaptureImage.Bitmap.Assign(Image);

  // take local copy of the file for processing
  s := TPath.GetDocumentsPath + PathDelim + 'photo.jpg';
  Image.SaveToFile(s);

  // asynchronously start image analysis
  aTask := TTask.Create (procedure ()
    var
      i: integer;
    begin
      if btnAn0.IsChecked then
        cv := ctAnalysis;
      if btnAn1.IsChecked then
        cv := ctOCR;

      if TMSFMXCLoudMSComputerVision1.ProcessFile(s, cv) then
      begin
        Description := '';

        if cv = ctAnalysis then
        begin
          // concatenate the image description returned from Microsoft Computer Vision API
          for I := 0 to TMSFMXCLoudMSComputerVision1.Analysis.Descriptions.Count - 1 do
          begin
            Description := Description + TMSFMXCLoudMSComputerVision1.Analysis.Descriptions[I] + #13#10;
          end;
        end
        else
        begin
          Description := TMSFMXCLoudMSComputerVision1.OCR.Text.Text;
        end;

        // update UI in main UI thread
        TThread.Queue(TThread.CurrentThread,
          procedure ()
          begin
            if Assigned(AnalysisResult) then
              AnalysisResult.Lines.Text := Description;
          end
          );
        TMSFMXCloudMSBingSpeech1.Connect;
      end
      else
      begin
        // update UI in main UI thread
        TThread.Queue(TThread.CurrentThread,
          procedure ()
          begin
            AnalysisResult.Lines.Add('Sorry, could not process image.');
          end
          );
      end;
    end
    );

  aTask.Start;
end;
In the TMSFMXCloudMSBingSpeech.OnConnected event, we can then send the text for speech synthesis to the Microsoft service and as a result we retrieve an audio stream that is then played through the device.

procedure TForm1.TMSFMXCloudMSBingSpeech1Connected(Sender: TObject);
var
  st: TMemoryStream;
  s: string;
begin
  st := TMemoryStream.Create;
  s := AnalysisResult.Lines.Text;
  try
    TMSFMXCloudMSBingSpeech1.Synthesize(s, st);
    TMSFMXCloudMSBingSpeech1.PlaySound(st);
  finally
    st.Free;
  end;
end;
So, it doesn't take much more than this to enhance the life of vision impaired people a little and let the iPhone read out what is around them or help with reading documents.

Now, let's try out the app in the real world. Here are a few examples we tested.

Using the app on the road to read road signs and capture car license plates

Trying to figure out what we see in a showroom

First the app analyzed correctly this is a bottle of wine in the cellar and is then pretty good at reading the wine bottle label.

You can download the full source code of the app here and have fun discovering these new capabilities.
Enjoy!

Bruno Fierens




This blog post has received 2 comments.


1. Friday, September 30, 2016 at 4:46:28 PM

Hello,
I have 2 questions:

1 - The bing speech return a "access denied" error
2 - Why only iphone works?

thanks
antonello

Carlomagno Antonello


2. Friday, September 30, 2016 at 4:49:38 PM

1) Did you obtain a proper API key for Bing speech?
2) We tested this sample for iPhone but I expect this to work for Android as well. I don''t see much reasons it wouldn''t work for Android in fact, but I haven''t tested it here so far on Android.

Bruno Fierens




Add a new comment:
Author:
Email:
  You will receive a confirmation mail with a link to validate your comment, so please use a valid email address.
Comment:
 
Change Image
Fill in the characters from the image above:
 

All fields are required.
 



TMS VCL Cloud Pack v3.7 introduces new level of seamlessness to access cloud data

Thursday, September 22, 2016

In TMS VCL Cloud Pack v3.7, new components TAdvmyCloudDataConnection and TAdvmyCloudDataDataSet have been added. With these 2 components, access to structured data on the cloud via the myCloudData service becomes easier than ever.

TAdvmyCloudDataConnection
TAdvmyCloudDataConnection is a non-visual component that manages access to myCloudData. It works as the intermediator for the authentication and authorization to access myCloudData for one or multiple TAdvmyCloudDataDataSet instances. This means that via the TAdvmyCloudDataConnection at least a one-time authentication and authorization is done with myCloudData to obtain the access token and the TAdvmyCloudDataDataSet can then work through TAdvmyCloudDataConnection to use this access token. To let the TAdvmyCloudDataDataSet use the connection is as simple as assigning the TAdvmyCloudDataConnection instance to TAdvmyCloudDataDataSet.Connection, just like we are used to assign a TADOConnection to a TADOTable for example.


TAdvmyCloudDataDataSet
TAdvmyCloudDataDataSet is a wrapper as a dataset of a table in the myCloudData service. When a connection is provided, set either the name of the table via the Table property or the unique ID of the table via the TableID property. When a connection has been made (i.e. access token was obtained), setting the property TAdvmyCloudDataDataSet.Active to true will fetch the table data from the myCloudData table and present it as a TDataSet through a TDataSource to any DB-aware component in VCL or via LiveBindings in FireMonkey applications. Note that this approach works both at run-time and design-time, so we can work with the cloud data at design-time to configure our DB-aware controls connected to it.


Dataset filter
An easy way is provided to perform basic dataset filtering. The TAdvmyCloudDataDataSet has a filter property which is presented as a collection of filter conditions. The filter condition consists of a field name, filter value, a filter condition logical operator and a filter comparison operator. The filter condition can be easily set at design-time where the dataset field can be picked and the values set.



From here we can work with the cloud data in much the same way from our code on the dataset as Delphi developers have been doing all the time with datasets, i.e. insert, delete, update records.

Rich metadata
But with the updated myCloudData service, there is much more than this possible as myCloudData now offers rich metadata for its structured data. This rich metadata can be programmatically accessed, created and updated but also easily configured via the myCloudData web interface. We'll cover this in a follow-up blog focused on how we can benefit from this rich metadata, but here you can already glance over the web interface you can access from your myCloudData account:


You can explore all this with a free myCloudData account. Create a new table via this account and you can use our TAdvmyCloudDataDataSetDemo application included in TMS VCL Cloud Pack to explore this.

Stay tuned for more!

Bruno Fierens




This blog post has received 2 comments.


1. Wednesday, September 28, 2016 at 12:29:39 PM

It''s wonderful... when will be avaliable for FXM Cloud pack?

Pallicera Xavi


2. Wednesday, September 28, 2016 at 12:31:25 PM

An update has just been released:
http://www.tmssoftware.com/site/forum/forum_posts.asp?TID=7020&title=ann-tms-fmx-cloud-pack-v3200-released


Nancy Lescouhier




Add a new comment:
Author:
Email:
  You will receive a confirmation mail with a link to validate your comment, so please use a valid email address.
Comment:
 
Change Image
Fill in the characters from the image above:
 

All fields are required.
 



TMS FNC UI Pack beta v1.5 available now for all TMS ALL-ACCESS customers!

Tuesday, September 06, 2016

We're pleased to inform TMS FNC UI Pack beta v1.5 is available now for all TMS ALL-ACCESS customers. In this beta version we've added no less than 10 NEW components! With TMS FNC UI Pack you only need to use 1 UI control set to master application development in VCL, FMX + LCL and target 5+ different operating systems.

What's new in the TMS FNC UI Pack beta release:

  • New: TTMSFNCGridDatabaseAdapter: adapter to connect the TTMSFNCGrid to any database
  • New: TTMSFNCToolBarPopup: popup version of the TTMSFNCToolBar
  • New: TTMSFNCRichEditorEditToolBarPopup & TTMSFNCRichEditorFormatToolBarPopup: popup version of the rich editor format and edit toolbar
  • New: TTMSFNCScrollBar: fully configurable scrollbar
  • New: TTMSFNCResponsiveList: responsive design list control
  • New: TTMSFNCSearchList: multi-column list with optional categorized items, pictures and text
  • New: TTMSFNCSearchEdit: editable version of the TTMSFNCSearchList
  • New: TTMSFNCHint: application / form wide HTML formatted hints
  • New: TTMSFNCURLBitmapContainer: bitmap container with the ability to download images from an URL
  • Comes with several new demos that demonstrate various features of the 10 new components we have added in this beta version.


We're eager to hear your experiences with the beta version!

A link from where the TMS FNC UI Pack beta 1.5 version can be downloaded has been added on the 'My products' page for all TMS ALL-ACCESS users. We welcome all comments, feedback & suggestions via email.

With TMS FNC UI Pack, you can target 3 frameworks and 5+ operating systems. Therefore we hope to receive as much feedback as possible about the use of these components on several platforms (VCL Win32/Win64, FMX Win32/Win64, MacOS-X, iOS, Android, LCL Win32/Win64, Mac OS-X, iOS, Android, Raspbian, ... ) and several IDE's (Delphi XE7, XE8, 10 Seattle, 10.1 Berlin, C++Builder XE7, XE8, 10 Seattle, 10.1 Berlin, Lazarus 1.4.4 or 1.6 with FPC 2.6.4 or FPC 3.0).

We look forward to all your feedback!

Nancy Lescouhier




This blog post has not received any comments yet.



Add a new comment:
Author:
Email:
  You will receive a confirmation mail with a link to validate your comment, so please use a valid email address.
Comment:
 
Change Image
Fill in the characters from the image above:
 

All fields are required.
 



September ... a busy 'event' month at tmssoftware

Thursday, August 25, 2016

We're pleased that in the month September we have again the chance to meet our customers, to show our products, to meet new developers who wishes to learn about our components & tools, ... on several upcoming events in Europe.

Bruno Fierens has also been invited as speaker, Bruno will talk about the challenges of creating cross-framework Pascal components. He will bring a session that discusses the differences between Pascal UI frameworks, how to build UI controls that can be used in the different UI frameworks and an introduction to the complex TMS FNC UI Controls that we have released so far. We will also show a sneak preview of a few brand new FNC UI controls that are currently in development.
An overview of the events:

SDN event - 2 September

Achmea Conference Center
Handelsweg 2, 3707 NH ZEIST
The Netherlands



Delphi Congrès 2016 - 15 September

FIAP
30 rue Cabanis, 75014 Paris
France



Delphi-Tage 2016 - 17 September

Mercure Hotel
Friesenstrasse 44-48, 50670 Köln
Germany



Pascal Delphi Festival 2016 - 20 September

Congrescentrum
Blokhoeve 1, 3438 LC Nieuwegein
The Netherlands



We're looking forward to see you at one of these events!

Nancy Lescouhier




This blog post has not received any comments yet.



Add a new comment:
Author:
Email:
  You will receive a confirmation mail with a link to validate your comment, so please use a valid email address.
Comment:
 
Change Image
Fill in the characters from the image above:
 

All fields are required.
 



From the team with love: the new TMS Component Pack 8.3

Thursday, July 14, 2016

We're excited to present the newest edition of the TMS Component Pack v8.3, our pack of over 400 VCL UI controls to make powerful, feature-rich Windows applications with Delphi & C++Builder.

The TMS Component Pack first version dates back from 1998 when our customers started asking for a bundle of our grid component and growing number of additional VCL UI controls developed by then.

TMS Component Pack is as such 18 years in the making and it's a relentless job to keep fine-tuning existing controls to changing needs, changing UI requirements, new UI paradigms & themes, new feature requests and add create brand new controls. Both surprising and exciting is that during all these 18 years, almost on a weekly basis, new ideas come in from you and from our team for component features and new components. A lot of our customers using TMS Component Pack UI controls are as passionate as our team about getting exactly the right look & feel, behavior and capabilities from the controls.

With this new version v8.3 we have added the usual list of new features, improvements and fixes. You can always consult this list here. One especially bigger extension was drag & drop support, sorting & filtering in our supercharged multicolumn treeview control. But in this blog, I wanted to draw your attention to new controls added in this release.

First of all, there is the new TAdvSearchList / TAdvSearchEdit.

The design of this component is inspired by new paradigms in search controls featured mostly on websites. Websites like Amazon, AliExpress, Google, Facebook, Booking.com, Trivago, ... all help the user in search by filtered suggestion lists as you type. TAdvSearchList is the basis for such list and offers a multicolumn list with optional categorized items, pictures, text and description with items, a place to show a number of search results and various criteria to find a match from first character, anywhere in a word, in text only or text and description etc...

Where the list control can be easily hooked up to a separat edit control and perform its search & filtering, the TAdvSearchEdit has the list embedded in a dropdown and it features additional optional buttons for directly filtering per categories when categories are used. It can as such be configured as a simple list to more complex and attractive looking lookup lists like this example:



Second new control added as is TAdvResponsiveList (preview).

The TAdvResponsiveList is a VCL UI control inspired by responsive design techniques and bringing this paradigm to Windows controls.As is the case with browsers, where we want to offer the best experience regardless of the browser client area size, Windows applications can also benefit from adapting the control's look & feel and behavior to the control's size to ensure the experience with your application is optimal regardless of the size of the screen of the user or regardless of a user running your application in a small window just in a corner of his desktop.



In a nutshell, TAdvResponsiveList offers a variable number of conditions that can be set for rendering its content depending on the size of the control. This can range from the number of columns depending on the width, the number or rows depending on the height, the absolute or relative size of cells when the control is resizing etc... As each cell offers rendering of HTML formatted text and this HTML formatted text can be the result of a template and in the conditions, variable templates can be set depending on the condition, i.e. size of the control. This way, more or less detail can be shown in the control item depending on its size.
But the sheer flexibility of this control becomes apparent when TAdvResponsiveList instances can be used as a child control in a TAdvResponsiveList and both parent and child can have their set of conditions. Or when you can create your own control descending from TAdvResponsiveList that can have fully custom drawn items.
A first developers guide to TAdvResponsiveList is available now. With TMS Component Pack 8.3, you can explore and start using the control from today in your applications. It is for now still marked as being in preview as this is all new to VCL application developers and we're eager to learn about your thoughts, comments, feedback, ideas & suggestions for next iterations of TAdvResponsiveList.

Bruno Fierens




This blog post has received 5 comments.


1. Thursday, July 21, 2016 at 7:03:52 PM

Good!
I hope to see them also in Firemonkey o FNC ...

Stefano Monterisi


2. Friday, July 22, 2016 at 2:43:40 PM

We''re actually already working on this.

Bruno Fierens


3. Tuesday, July 26, 2016 at 9:56:43 AM

Great. I would love to see a FMX version.

Kouraklis John


4. Thursday, July 28, 2016 at 2:57:11 PM

Thanks a lot for the TAdvSearchList/-Edit components! I''ve been waiting for something like that for some time... ;-)

PS: When I search for "TAdvSearchList" on your site the search result is empty. Doesn''t your search include the blog entries? Would be helpful if it did.

Andreas Friedrich


5. Friday, July 29, 2016 at 8:28:30 AM

TAdvSearchList does not have its own separate page, it''s planned to add that and I suspect it should then turn up in search.

Bruno Fierens




Add a new comment:
Author:
Email:
  You will receive a confirmation mail with a link to validate your comment, so please use a valid email address.
Comment:
 
Change Image
Fill in the characters from the image above:
 

All fields are required.
 




Previous  |  Next  |  Index