Is it possible for users to specify their own modified amino acids (outside the list supported by HADDOCK) and have them treated like a ligand (but still remain attached to rest of the protein) during docking?
This is not possible - at best you could defined those as HETATM but they will not be connected to the adjacent amino acids (you could impose a few distance restraints to keep them in place…)