Possible Concurrency issue in FixInsight

DEV TMS FixInsight Pro
1
Hi Folks,

on our Jenkins server, we are using build pipelines with parallel steps to speed up the build. Every now and then FixInsight 2016.09 fails with the following error:
It fails on different projects, but always at the same address. Could this be a concurrency issue? I'm pretty sure it is, because the issue only showed up after I started to let all compile and FixInsight tasks run in parallel. There are ~10 compile and ~10 FixInsight task running at the same time for one build job in a Windows 10 VM. There may be more than one build job running at the same time.

All FixInsightCL tasks are using the same .ficfg file, but different .dpr files, which may share a lot of other source files. Are you locking files exclusively while analysing or loading them? (.ficfg, .pas, .dpr, ...)

2

It doesn't need to lock anything and it shouldn't. But I'll double check, I've missed something perhaps. Thanks for letting me know!

3
Check all your TFileStream.Create(FileName, fmOpenRead) statements. Under the hood it seems to be (.. or fmShareExclusive) if you don't specify anything.
We ran into this issue with a VCL styled application, which sometimes crashed when started in parallel. The crash happened because of the following line in Vcl.Themes.TStyleManager.LoadFromFile

In Berlin it was:
    LStream := TFileStream.Create(FileName, fmOpenRead);

In Tokyo it now is:
    LStream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);

4
Did you find something that could have triggered the possible concurrency issue? If yes, does the update address this problem? I tried working with a copy of the .ficfg file for each instance of FixInsightCL. This improved the situation, but didn't fix it. There are still build failures (one out of ten) with the
exception. So FixInsight possibly also locks .pas files that are shared across different projects
5

In the latest build I explicitly use fmOpenRead when reading .pas files. I can't prove if it helps, it it may. Please give it a try.

6

Thanks for the Info. I think you have to specify fmOpenRead or fmShareDenyWrite.
fmOpenRead alone doesn't seem to be enough as I said in the example above.

7

I fear that the concurrency problem still exists, but now it manifests itself in a different way. I let a build run on our server with 10 dcc32 and 10 FixInsightCL tasks in parallel.
The first one failed on one project with lots of "Fatal parser error" messages in line 1 col 1 for 10-12 different .pas files.

<message line="1" col="1" id="FATAL">Fatal parser error</message>

I triggered a few more builds: Same failure but with a different .dpr and different .pas files this time.

The interesting thing is that with 2016.09 and 2017.11 I only had intermittent "Exception EInOutError in module FixInsight.Services.dll at 000B3B7C" failures. With 2019.01upd I now have 100% failures in the parallel analysis step.

Currently I run all FixInsight and all dcc32 tasks in parallel (10+10 at the same time). Before I tried to optimize the parallel builds I ran only two parallel tasks. One for dcc32 which built 10 applications and one for FixInsight which ran the analysis on the same 10 applications.
Since FixInsight is the most time consuming step of our build, I really want to cut down the time.

I'll now try two things:
1. run all FixInsight steps sequential, but in parallel to dcc32. This will probably tell me, if FixInsight interferes with dcc32.
2. run all FixInsight steps parallel but in sequence to all parallel dcc32 steps. So FixInsight and dcc32 should not interfere with each other.

I have to mention that FixInsight 2019.01upd1 is not installed "properly" on the build server. The installed version is still 2016.09 and FixInsight2019.01 is run from a folder which contains just the .exe and .dll files.

Everything runs fine locally with 2019.01upd1 installed, but there are no parallel tasks. Maybe I write a script that fires 10xCreateprocess or so.

8
Some feedback:
1. running the FixInsight steps themselves serial but parallel with dcc32 causes the fatal parser errors
2. running the FixInsight steps parallel but before or after dcc32 doesn't cause the fatal parser errors

So it looks like dcc32 locks the .dpr/.pas files, but it didn't show up until now.
9

Fun fact: several instances of dcc seem to have no problem running in parallel

10
Very interesting, thank you!

But how can we solve this... Do you think it worth trying fmOpenRead or fmShareDenyNone ?
11
If you use fmOpenRead or fmShareDenyNone, you allow anything to happen to the file that you are currently reading. I think this is too dangerous.
fmOpenRead or fmShareDenyWrite would allow other applications to read the file as too and as long as they are well behaved and do not lock the file exclusively, everything is fine.

The EInOutError exception that we had before was a better error than the 'Fatal parser error' because it gave me a hint that the file could be locked by another process. The 'Fatal parser error' could be caused by anything unless you manage to log the reason as well.
12
Hi Roman,
I still get the occasional "Fatal parser error" on our build system. So the concurrency problem seems to persist. What happens on your if you start FixInsightCL for the same project in several dos shells at the same time?
13

Finally I was able to reproduce that. Seems like fmShareDenyWrite was missing at one place. I'll upload an update tomorrow.

14

Please try the update http://sourceoddity.com/download/FixInsight_2019.01upd1_setup.exe

Hopefully the issue is fixed...

15
Hi Roman,

thanks and sorry for the long silence. I just went on vacation when you posted these mails. Will try the new version immediately and report back.
16
I tried it out, but reverted to the release version, because this update introduced unexpected W525 "Missing INHERITED call in constructor" warnings, which caused our builds to fail.
I will describe this in a separate topic.