For Windows XP, normal operation may not be guaranteed because Microsoft has ended support for this operating system. (Reference)

This issue occurs because the program you are trying to record is using the overlay mixer. 

The easiest way to solve the problem is as follows.

Right-click on the desktop and click Properties.

The Display Properties window will appear. Click the Settings tab.

Click the Advanced button.

Click the Troubleshoot tab.

As shown below, Hardware acceleration is set to Full. Change it to None.

After setting it as shown below, click OK, close the program you want to record, and then try recording again.