1660ti if you want the Encoder, 1080ti for raw performance. RX580 is not worth getting vs either of the other 2.
the 1660ti has New NVENC which is an amazing encoder but the 1080ti should beat it in performance. The RX 580 is slower than the 1660ti and 1080ti along with its Encoder being worse than both.The Encoder on Polaris in linux can barely handle 1080p 30fps and itll still block and pixelate regardless of bitrate. Kernel drivers not worth it in that case.
They are all decent cards for all around use but definitely either one of the Nvidia should be good. I have an RX 590 which is a great card. It’s right there with my GTX 1060. I think the 1080TI would be a good card if you feel it’s worth what you are paying for it and you think it will be problem free.
@tnthomas
What software are you going to be using to do what you are intending to do?
About a year ago I began dabbling in editing, we had the idea of using a green screen, and shot videos with it for a couple months. I was able to add a tropical themed photo as background. I don’t remember precisely which app I used, but I do recall having experimented with DaVinci Resolve and Kdenlive. The encoding took some time, I think I used my Dell laptop, which have a GTX 1050, none of my desktop’s GPU could handle much more than providing a display.
I’m really gravitating towards the idea of doing a GPU upgrade, not just for the Xeon desktop, but for my ‘daily’ desktop, the GA-78LMT-S2 | Phenom II X4 965 | MSI 8400GS | 8GB DDR3. I love the Phenom II X4 965 and don’t want to retire it, but in recent times have experienced some freeze-ups while browing websites that are ‘code-heavy’. I did a thread here on that subject, the matter still is uncertain, but all hardware checks out OK, and I’m thinking the 8400GS may be the weak-link.
So, I am of the mind to do GPU upgrades on both the GA-78LMT-S2 rig and the Xeon desktop, and foregoing a complete new system build. In addition to an upgraded GPU, additional RAM could be a plus, particularly since RAM is looking more affordable now, than the last time I shopped.
So, those two machines each have an antiquited 8400GS, and I can justify upgrading both, so buying both the GeForce GTX 1660 Ti and the XFX Radeon RX 580 GTS(or a 2nd Nvidia card) would be an easy, fun project that would breathe new life into two machines that I love.
It probably would be advisable to remove your existing fan and heatsink and reapply new high grade thermal paste to your cpu after cleaning off the original stuff since they are older.
It wouldn’t hurt, the Xeon’s been sitting in the socket since 2011. The GA-78LMT-S2 Socket AM3 got upgraded from a AMD Athlon II X2 260 to the Phenom II X4 965 last summer, so it’s themal paste is fresh.
I have the GeForce GTX 1660 Ti sitting in my Newegg cart, haven’t decided about the 2nd GPU, which would be going in the AM3 machine.
What’s in it the AM3 machine now?
Edit: Looked at previous post and see it’s the 8400GS.
Edit2: Do you have that GPU in both of your machines?
Yes, bought them close to the same time off ebay for < $20. Pretty sad, huh? Folding@Home was my “gaming” and my passion since 2002. The most powerful GPU I ever owned was a GTX 970, should have neither sold it. I was going to upgrade but procrastinated. In the meantime the crypto mining thing got going, drove up GPU prices and I’ve been waiting for them to come down ever since. 
I will never pay over MSRP and I have seen double and triple prices by unscrupulous vendors. I only buy from reputable sources of computer parts that don’t do this. Supply has been the biggest issue although prices have risen. That is a given due to events of the last two years. I have limits on what i will spend and what i feel is reasonable and also value for my money.
Edit: The Gigabyte 1660 TI here is in stock where i buy my parts. They have 5 in stock locally. @499.99 Cdn + taxes.
I would very much recommend Davinci Resolve or Lightworks as Linux editors. Flowblade and Kdenlive are great OSS alternatives but Davinci and Lightworks are hard to beat and you can use both for free.
This is going to be the case for many systems around this era. Modern web is much heavier and on Linux GPU acceleration for web browsing isn’t as good as windows so its VERY CPU reliant. I’m not confident a repaste would help as the old Phenom II x4 920 system I have with an R7 260 GPU is a similar experience and the 260 is 100x faster than the 8400GS lol.
No one said a re-paste is going to help or make any difference. I just recommended it based on the age of the systems although i didn’t know the Phenom II x4 965 was a recent upgrade. It only makes sense if the system is a decade old and hasn’t been touched and the OP wishes to maintain this system for some time to come. It’s not a bad idea. 
Repaste is fine, never really hurts I’m just saying his poor web experience is likely the age of the system as I own a similar old system I tinker with and its got a similar experience. Sometimes you reach a point with maintaining an old system its just on life support lol.
Yes, this issue was a big part of my new build motivation. Plus, those 8400gs cards have towed the line on display duty long enough. When I get the GTX 1660 Ti of course I’ll have to see how well she does F@H.
Update:
Well, a Folding@home team member made me an offer I couldn’t refuse, an EVGA 1080ti for $380, so I bought it. It is capable of better video editing performance that the 1660ti, and is costing me about $50 less. It is used, but well taken care of, and I trust the seller as we’ve been f@h team-mates for 17 years.
I’ll do some folding on it for a couple months 24/7, but I can’t really afford the increased electric bill. After that, it will mostly serve as a ‘display’ card and modest video editing duty.
The 1080ti will go in the Xeon E2650 machine and the AM3 rig will probably get the 1660ti card.
Sounds like a plan. 