Alright turns out it's Optifine for some reason. With Schematica loaded but Optifine not I managed to connect. Also with just Optifine and Forge I couldn't connect, but just Optifine I can. Weird