OpenCPPCover with native C++ test

Jun 6, 2016 at 10:23 AM
Edited Jun 6, 2016 at 10:25 AM
Hi,

I'm attempting to get OpenCPPCover to work with a native C++ test using Visual Studio Community 2015. Layout of the application:
  • Engine.lib (static library project)
  • UnitTests.dll (native C++ unit test project)
I'm attempting to run the unit tests using:

C:\Projects\Engine\x64\Debug>opencppcoverage --continue_after_cpp_exception --cover_children --modules C:\Projects\Engine --sources C:\Projects\Engine -- "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /Platform:x64 UnitTests.dll

The results:
  • Unit tests execute fine;
  • The report outlines the correct files;
  • Coverage is not measured. Every file gets 0% covered, 100% uncovered (even though I know that's not correct).
Omitting 'sources' and 'modules' command line options don't change the results. For the relevant DLL's, I see only information messages in the output ([...] is selected because it matches selected pattern: *).

The DLL is compiled as x64 DLL and I'm using version 0.9.5.2 (x64).
Jun 6, 2016 at 10:30 AM
More information: I just tested an independent EXE file to be sure.

It appears like the problem is with the vstest.console.exe application; normal exe files do give coverage results.
Coordinator
Jun 7, 2016 at 9:44 AM
Hi,

I already tested if vstest.console.exe works with OpenCppCoverage here.

I have just tested again with OpenCppCoverage 0.9.5.2 with and without Platfrom:x64 and have the correct coverage in both case.
I can reproduce an empty coverage when the project is managed (with [TestClass] and [TestMethod]). Is your library a managed dll?

If it is not the case, can you provide a simple project where you can reproduce this issue so that I can fix it? (You need to create an issue if you want to upload files)

Let me know if it solves your problem,

OpenCppCoverage
Jun 7, 2016 at 10:40 AM
Hi,

I can confirm that my project is a native, unmanaged, non-clr DLL library.

Yesterday I've also modified my unit tests a bit, so that I can call them from my own C++ test/EXE application (without using vstest.console.exe; it simply uses LoadLibrary and self-registering functions).

It is interesting that you mention managed code. The vstest.console.exe does load a lot of managed DLL's in the process, as you can see in the log file: https://data.nubilosoft.com/opencppcoverage_log.txt .

Even more annoying is that I've attempted to make a minimum test case, which should work exactly the same as my engine DLL ( https://data.nubilosoft.com/TestCPPCoverage.7z ) -- but unlike the Engine, this one actually works fine with OpenCPPCoverage. That's not making it easier...

Please let me know if you spot anything out of the ordinary; otherwise I'd be happy to put some breakpoints in the OpenCPPCoverage code myself; some pointers on where to look are much appreciated though...

Cheers,
Stefan.

PS: some time ago I created a visual studio addin for highlighting managed code coverage results. If this works, I'd be happy to share that code, so we can make a fully integrated addin for VS.
Coordinator
Jun 8, 2016 at 9:06 AM
Hi,

I do not spot anything special in the log file. It is possible to post the log file with --verbose flag?

I suggest simplifying your code while you can reproduce your problem. I think it is easier than debugging the code. Verbose log can probably help me to give you some pointers if you cannot simplify your code.

Thank a lot for proposing your help for the plugin, I really appreciate, but I am already working on improving the UI for the plugin and another people already proposed to add the colors for cover uncover lines directly in the editor.

Hope that help,

OpenCppCoverage
Jun 8, 2016 at 2:29 PM
Hi there,

Simplifying code is easier said then done if you have a piece of production code with hundreds of thousands of lines of code at your hands... If I could have made a minimum test case, I definitely would have...
Verbose log can probably help me to give you some pointers if you cannot simplify your code.
I posted the verbose output: https://data.nubilosoft.com/opencppcoverage_verbose.txt .
Thank a lot for proposing your help for the plugin, I really appreciate, but I am already working on improving the UI for the plugin and another people already proposed to add the colors for cover uncover lines directly in the editor.
Yeah... I kinda needed it today, so I took the liberty of implementing it based on what I had. All things considered, I believe it's pretty much what people would need...

You can find the code at https://github.com/atlaste/VSOpenCPPCoverage .
Coordinator
Jun 11, 2016 at 10:47 AM
Hi,

I read carefully the verbose log but do not see anything special.
I also tried to perform several additional tests but was unable to reproduce your issue.

By reading the log, I can tell that OpenCppCoverage registered correctly some lines for coverage ("Executable lines for").
You can check that you have a line starting by "Executable lines for" where you expect to have a coverage.

For pointers in the code, the method ExecutedAddressManager::MarkAddressAsExecuted is called when hitting a selecting line and a good start for debugging.
ExecutedAddressManager::CreateCoverageData is where the coverage is created.

Note: Compile OpenCppCoverage can be a little tricky and there is no documentation. You can find information here

Thanks for the plugin code, I will have a look.

Let me know if you find something.

OpenCppCoverage
Jun 23, 2016 at 11:51 AM
Edited Jun 23, 2016 at 11:51 AM
Hi there,

Sorry for the late reply, I've been a bit sick.

Today I compiled the code and put some breakpoints in there as you suggested on the points outlined here (on #1 and #2)
    auto it = addressLineMap_.find(address); // #1

    if (it == addressLineMap_.end())
        return boost::none;

    auto& instruction = it->second; // #2
I can see that #1 is hit; to be exact, addressLineMap_ contains approximately 1 million breakpoints for my project. However, #1 is hit only 3 times, and #2 is never hit. During this time, I can see in the console window that the tests are being executed.

I also find it interesting that this behavior only occurs when running through vstest. When I run my workaround (load & execute a native DLL) there are no problems. Is it possible that the issue lies somewhere in the child-processes binding? Any way I can check this?

Or perhaps any other pointers where to look next?

Cheers,
Stefan.
Coordinator
Jun 29, 2016 at 10:40 PM
Edited Jun 29, 2016 at 10:45 PM
Hello,

From your analysis, OpenCppCoverage seems to be able to register correctly the lines (addressLineMap_ contains a lot of items) but there is no (or wrong) even when a line is executed. This is the same behavior as described in the previous log you sent.

It is hard to find a solution without more knowledge about your code but you can try the following ideas:
  • You can put a breakpoint inside Debugger::OnException. This function is called for each executed line and for every thread creation. If this function is called often, you can trace and see why #1 is not called.
  • Another idea is to call DebugBreak in a location you expect to be monitored. If you run with OpenCppCoverage, you should see the message "It seems there is an assertion failure or you call DebugBreak() in your program."
  • You can also call IsDebuggerPresent in a location you expect to be monitored. If you run with vstest.console.exe from command line without OpenCppCoverage this should return false.
  • Check if your dll has a special settings under properties C++ or linker compare to a default dll.
It is perhaps related to the child process binding. I will try to make a more complex test with vstest.console.exe to see if I can find something.

Hope that help and let me know if you find a solution,

OpenCppCoverage
Coordinator
Jun 30, 2016 at 8:28 PM
Hello,

You can also try the following things:
  1. Make a copy of your project
  2. Add a new test with one line #1
  3. Check if OpenCppCoveage produces coverage for test #1 (You probably need to update --sources and --modules)
    3.1. if yes: You can chose another test and comment code until you see coverage for this test.
    3.2. if no: You can remove all your tests but keep #1.
    3.2.1. If you can see coverage for #1, there is probably a function call that create the problem (dichotomy should help)
    3.2.2. If you cannot see the coverage, you can remove the link with your static dll and try again:
    3.2.2.1. If this solve the problem, there is probably a special settings in the static library
    3.2.2.2. If not, that means you have a dll with a single test case that does not work. You can compare the dll settings with the default one.
Let me know if you find something,

OpenCppCoverage
Jul 6, 2016 at 12:36 PM
Edited Jul 6, 2016 at 12:36 PM
I finally found the time to do some more tests.

I first tried the last post that you had:
  1. Made a copy of the project to another location
  2. Removed tests until I had only 1 left
  3. Coverage is generated.
    3.1. Copied more tests... it all seems to be working here.
I then restored the referenced libraries, include paths, etc - all working fine. However, as I said, I tested it on a different location on disk, which obviously matters because of the sources filters. I'm not using a modules filter anymore. Having this different path means that the process doesn't need to bind breakpoints to the engine.

As I said before, the small test cases are working just fine. The problems start to occur when I'm attempting to run coverage for my engine project (which is huge).

Next, I tried the suggestions in the first post:
Check if your dll has a special settings under properties C++ or linker compare to a default dll.
No difference here with a smaller test, I checked all properties.
Another idea is to call DebugBreak in a location you expect to be monitored. If you run with OpenCppCoverage, you should see the message "It seems there is an assertion failure or you call DebugBreak() in your program."
Confirmed, this happens.
You can put a breakpoint inside Debugger::OnException. This function is called for each executed line and for every thread creation. If this function is called often, you can trace and see why #1 is not called.
What I observe is that CodeCoverageRunner::OnException will return FirstChanceException -> NotHandled each time it's called. On closer inspection, ExceptionHandler::HandleException attempts to find the exceptionCode in breakPointExceptionCode_, which is never present (the dictionary contains only 2 codes, which I understand to be related to breakpoints)

The exception code that occurs the most is 0xE06D7363, which seems to be a SEH exception ( https://support.microsoft.com/en-us/kb/185294 ). To make it even stranger, when I filtered that code out, it doesn't seem to generate any exception codes anymore. I believe this can only mean that debug events from the child process aren't captured for some strange reason...

So, I've turned my attention to de Debugger class next and compared it to the information in (sysinternals) process explorer. Both processes (vstest.console.exe and TE.ProcessHost.Managed.exe) are present in the processHandles dictionary. The relevant threads (WEX.Common...) are also present in the threadHandles dictionary.

It's strange... I understand that if breakpoint events aren't captured for some reason, it won't work. Still, the events show that it should work.
You can also call IsDebuggerPresent in a location you expect to be monitored. If you run with vstest.console.exe from command line without OpenCppCoverage this should return false.
Hmm... you say something interesting here.

I'm using custom exceptions, which also generate stack traces through DbgHelp. To be on the safe side, I tested if it interferes with a minimum test... it seems not to interfere, but better safe then sorry. Here's the relevant test code:
HANDLE process = GetCurrentProcess();
SymSetOptions(SYMOPT_LOAD_LINES);
SymInitialize(process, NULL, TRUE);

static const int MaxCallStack = 62; // // ## The sum of the FramesToSkip and FramesToCapture parameters must be less than 63 on Win2003
void* callers[MaxCallStack];
auto stackCount = RtlCaptureStackBackTrace(1, MaxCallStack, callers, NULL);

SYMBOL_INFO* symbol = reinterpret_cast<SYMBOL_INFO *>(calloc(sizeof(SYMBOL_INFO) + 256 * sizeof(char), 1));
symbol->MaxNameLen = 255;
symbol->SizeOfStruct = sizeof(SYMBOL_INFO);

SymFromAddr(process, reinterpret_cast<DWORD64>(callers[0]), 0, symbol);

Either way, I've checked with your suggestion as well. IsDebuggerPresent returns false without opencppcoverage.

To be entirely thorough I've also checked this WITH opencppcoverage and vstest. IsDebuggerPresent returns true here.

Last but not least, even though I always want everything to work and it kinda feels unsatisfactory, I'm on the virge of giving this this one up. The workaround that I've created earlier works great and is much, much faster than running everything though vstest; to be honest it's very unlikely that I'll move back to the vstest route.

PS: have you already looked at the VS plugin I wrote?
Coordinator
Jul 11, 2016 at 11:22 AM
Hello,

You did a very depth analysis of the problem.
The exception 0xE06D7363 is the SHE code for C++ exception and can explain your problem.

I tested the code you post but did not find something.

I tried to create a more complex using vstest.console.exe. I am able to reproduce the same issue (No coverage).

I tested your VS plugin. You made an impressive work in a few days...
I am able to see the coverage for VS unit tests without any problem.

When I tested on normal executable, I had an issue.
In CoberturaData, line 106 lookup.Add(file.ToLower(), current); I get exceptions because the key already exist. I did a quick hack by putting a "try catch" around line 106 and was able to see coverage.
Note: I tested on a computer without Visual Studio 2013 and so I updated all versions of the dll from v12 to v14.

I also noticed that "CoverageReport windows" is always empty in my case.

Note that I am also working on improving the current plugin. I plan to make a completely new release for the end of the summer.

I am going to update this discussion as soon as I have more information about the empty coverage issue.

OpenCppCoverage
Jul 11, 2016 at 1:54 PM
I tried to create a more complex using vstest.console.exe. I am able to reproduce the same issue (No coverage).
Ah that's good news! Now we're getting somewhere.
I tested your VS plugin. You made an impressive work in a few days...
Thanks.
When I tested on normal executable, I had an issue.
I've just checked in a newer version that has this bug fixed. I also updated the Visual Studio version numbers; I'm running it on VS2015 myself, but should also work on other versions now. It's interesting that the coverage report window is empty though...
Note that I am also working on improving the current plugin. I plan to make a completely new release for the end of the summer.
Well, as I've said, perhaps it would be best to make a combination. I'd be happy to improve the VS addin that I've shared as well.
Coordinator
Jul 13, 2016 at 8:28 PM
Hello,

I have just found the issue:
  • vstest.console.exe loads XXX.dll
  • OpenCppCoverage adds some breakpoints and fills addressLineMap_ in ExecutedAddressManager::RegisterAddress
  • vstest.console.exe unloads XXX.dll => Every breakpoints are lost
  • vstest.console.exe loads XXX.dll
  • OpenCppCoverage does not add breakpoints because they already exist in addressLineMap_
    => There is no coverage
As a quick hack, you can replace in ExecutedAddressManager::RegisterAddress, if (itAddress == addressLineMap_.end()) by if (instructionValue != 0xCC). It should solve the problem temporary.

I am going to develop a real fix by handling dll unload correctly.

Thanks a lot for all the information you provide to help solving this issue!!!!

OpenCppCoverage
Coordinator
Jul 27, 2016 at 9:39 PM
Hello,

OpenCppCoverage 0.9.5.3 is available and should solve the problem.

Let me know if it is not the case,

OpenCppCoverage
Marked as answer by OpenCppCoverage on 8/20/2016 at 8:12 AM
Aug 2, 2016 at 9:46 AM
Hello,

I find issue in viewing the index.html file logged after executing RunOpenCppCoverage. I am using version OpenCppCoverage 0.9.5.3. It just displays Coverage total lines Items and rest is empty. Can you please tell me if I have to do any settings in my VS2013 pro.
Coordinator
Aug 20, 2016 at 4:17 PM
Hello Sowmyashree,

My answer is in the issue you created: https://opencppcoverage.codeplex.com/workitem/21

OpenCppCoverage