Windows and Game huge FPS drops

So, a couple of days I go I bought a Zotac 980Ti. I used to run on a MSI 780Ti, but needed a upgrade. I plugged my main screen into the the 980Ti (HDMI) and the rest of the screens into the 780Ti. I set the 780Ti card as a PhysX dedicated card. After bootup I quickly noticed that something was wrong. Whenever I was playing a YouTube video the general FPS of everything that happened on my PC went down. Moving windows on the screen was not smooth at all, and not enjoyable. I tried launching a game, and the FPS is not how a 980Ti should run it. The game I tried was Elder Scrolls Online, which is a not too demanding game. The frames there jums from 60 to 5 all the time, and everything between. I then tried to disable the PhysX dedication, and it did no difference. I also tried to restart the PC with the secondary card disabled, and this fixed the problems, but I need all 3/4 screens, and the 980Ti only have 1 HDMI output.. Also, the Start menu on Win8 lags whenever I open it or close it, even if NOTHING is running on the PC (videos or games or any programs). The PC got a fresh reinstall of Windows when I installed the new card, so there is no malicious software on the PC.

Does anyone know how this can be fixed?

PC Specs:

CPU: Intel i7-3820
GPU: Nvidia GTX980Ti/GTX780Ti
RAM: 32GB HyperX Beast
M/B: MSI Big Bang XPowerII
Windows runs on SSD