A parallel implementation of a quasi-static Partial Element Equivalent Circuit (PEEC)-based solver that can handle electromagnetic problems with non-orthogonal structures is presented in this paper. The solver has been written in C++ and employs GMM++ and ScaLAPACK computational libraries to make the solver fast, efficient, and adaptable to current parallel computer systems. The parallel PEEC-based solver has been tested and studied on high performance computing clusters and the correctness of the solver has been verified by doing comparisons between results from orthogonal routines and also another type of electromagnetic solver, namely FEKO. Two non-orthogonal numerical test cases have been analysed in the time and frequency domain. The results are given for solution time and memory consumption while bottlenecks are pointed out and discussed. The benchmarks show a good speedup which gets improved as the problem size is increased. With the capability of the presented solver, the non-orthogonal PEEC formulation is a viable tool for modelling geometrically complex problems.