Web forum is in read-only mode. Login as active registered customer for write access
  Forum Search   New Posts New Posts

Possible Concurrency issue in FixInsight

 Post Reply Post Reply
Author
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Topic: Possible Concurrency issue in FixInsight
    Posted: 12 Feb 2019 at 1:22pm
Hi Folks,

on our Jenkins server, we are using build pipelines with parallel steps to speed up the build. Every now and then FixInsight 2016.09 fails with the following error:
Exception EInOutError in module FixInsight.Services.dll at 000B3B7C
It fails on different projects, but always at the same address. Could this be a concurrency issue? I'm pretty sure it is, because the issue only showed up after I started to let all compile and FixInsight tasks run in parallel. There are ~10 compile and ~10 FixInsight task running at the same time for one build job in a Windows 10 VM. There may be more than one build job running at the same time.

All FixInsightCL tasks are using the same .ficfg file, but different .dpr files, which may share a lot of other source files. Are you locking files exclusively while analysing or loading them? (.ficfg, .pas, .dpr, ...)

Back to Top
Roman Yankovsky View Drop Down
New Member
New Member
Avatar

Joined: 21 Jun 2016
Posts: 26
Post Options Post Options   Quote Roman Yankovsky Quote  Post ReplyReply Direct Link To This Post Posted: 13 Feb 2019 at 10:12pm
It doesn't need to lock anything and it shouldn't. But I'll double check, I've missed something perhaps. Thanks for letting me know!
Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 15 Feb 2019 at 11:32am
Check all your TFileStream.Create(FileName, fmOpenRead) statements. Under the hood it seems to be (.. or fmShareExclusive) if you don't specify anything.
We ran into this issue with a VCL styled application, which sometimes crashed when started in parallel. The crash happened because of the following line in Vcl.Themes.TStyleManager.LoadFromFile

In Berlin it was:
    LStream := TFileStream.Create(FileName, fmOpenRead);

In Tokyo it now is:
    LStream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);

Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 9:47am
Did you find something that could have triggered the possible concurrency issue? If yes, does the update address this problem? I tried working with a copy of the .ficfg file for each instance of FixInsightCL. This improved the situation, but didn't fix it. There are still build failures (one out of ten) with the
EInOutError in module FixInsight.Services.dll at 000B3B7C

exception. So FixInsight possibly also locks .pas files that are shared across different projects
Back to Top
Roman Yankovsky View Drop Down
New Member
New Member
Avatar

Joined: 21 Jun 2016
Posts: 26
Post Options Post Options   Quote Roman Yankovsky Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 10:15am
In the latest build I explicitly use fmOpenRead when reading .pas files. I can't prove if it helps, it it may. Please give it a try.
Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 11:12am
Thanks for the Info. I think you have to specify fmOpenRead or fmShareDenyWrite.
fmOpenRead alone doesn't seem to be enough as I said in the example above.

Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 11:27am
I fear that the concurrency problem still exists, but now it manifests itself in a different way. I let a build run on our server with 10 dcc32 and 10 FixInsightCL tasks in parallel.
The first one failed on one project with lots of "Fatal parser error" messages in line 1 col 1 for 10-12 different .pas files.

<message line="1" col="1" id="FATAL">Fatal parser error</message>

I triggered a few more builds: Same failure but with a different .dpr and different .pas files this time.

The interesting thing is that with 2016.09 and 2017.11 I only had intermittent "Exception EInOutError in module FixInsight.Services.dll at 000B3B7C" failures. With 2019.01upd I now have 100% failures in the parallel analysis step.

Currently I run all FixInsight and all dcc32 tasks in parallel (10+10 at the same time). Before I tried to optimize the parallel builds I ran only two parallel tasks. One for dcc32 which built 10 applications and one for FixInsight which ran the analysis on the same 10 applications.
Since FixInsight is the most time consuming step of our build, I really want to cut down the time.

I'll now try two things:
1. run all FixInsight steps sequential, but in parallel to dcc32. This will probably tell me, if FixInsight interferes with dcc32.
2. run all FixInsight steps parallel but in sequence to all parallel dcc32 steps. So FixInsight and dcc32 should not interfere with each other.

I have to mention that FixInsight 2019.01upd1 is not installed "properly" on the build server. The installed version is still 2016.09 and FixInsight2019.01 is run from a folder which contains just the .exe and .dll files.

Everything runs fine locally with 2019.01upd1 installed, but there are no parallel tasks. Maybe I write a script that fires 10xCreateprocess or so.

Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 3:53pm
Some feedback:
1. running the FixInsight steps themselves serial but parallel with dcc32 causes the fatal parser errors
2. running the FixInsight steps parallel but before or after dcc32 doesn't cause the fatal parser errors

So it looks like dcc32 locks the .dpr/.pas files, but it didn't show up until now.
Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 3:55pm
Fun fact: several instances of dcc seem to have no problem running in parallel
Back to Top
Roman Yankovsky View Drop Down
New Member
New Member
Avatar

Joined: 21 Jun 2016
Posts: 26
Post Options Post Options   Quote Roman Yankovsky Quote  Post ReplyReply Direct Link To This Post Posted: 20 Feb 2019 at 5:06pm
Very interesting, thank you!

But how can we solve this... Do you think it worth trying fmOpenRead or fmShareDenyNone ?
Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 21 Feb 2019 at 10:46am
If you use fmOpenRead or fmShareDenyNone, you allow anything to happen to the file that you are currently reading. I think this is too dangerous.
fmOpenRead or fmShareDenyWrite would allow other applications to read the file as too and as long as they are well behaved and do not lock the file exclusively, everything is fine.

The EInOutError exception that we had before was a better error than the 'Fatal parser error' because it gave me a hint that the file could be locked by another process. The 'Fatal parser error' could be caused by anything unless you manage to log the reason as well.
Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 04 Mar 2019 at 1:35pm
Hi Roman,
I still get the occasional "Fatal parser error" on our build system. So the concurrency problem seems to persist. What happens on your if you start FixInsightCL for the same project in several dos shells at the same time?
Back to Top
Roman Yankovsky View Drop Down
New Member
New Member
Avatar

Joined: 21 Jun 2016
Posts: 26
Post Options Post Options   Quote Roman Yankovsky Quote  Post ReplyReply Direct Link To This Post Posted: 06 Mar 2019 at 10:49pm
Finally I was able to reproduce that. Seems like fmShareDenyWrite was missing at one place. I'll upload an update tomorrow.
Back to Top
Roman Yankovsky View Drop Down
New Member
New Member
Avatar

Joined: 21 Jun 2016
Posts: 26
Post Options Post Options   Quote Roman Yankovsky Quote  Post ReplyReply Direct Link To This Post Posted: 07 Mar 2019 at 10:37pm

Please try the update http://sourceoddity.com/download/FixInsight_2019.01upd1_setup.exe

Hopefully the issue is fixed...

Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 21 Mar 2019 at 11:40am
Hi Roman,

thanks and sorry for the long silence. I just went on vacation when you posted these mails. Will try the new version immediately and report back.
Back to Top
Onken L. View Drop Down
Member
Member
Avatar

Joined: 16 Jun 2010
Posts: 60
Post Options Post Options   Quote Onken L. Quote  Post ReplyReply Direct Link To This Post Posted: 21 Mar 2019 at 2:10pm
I tried it out, but reverted to the release version, because this update introduced unexpected W525 "Missing INHERITED call in constructor" warnings, which caused our builds to fail.
I will describe this in a separate topic.
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down