Thanks for posting, and thanks for the quick follow-up letting us know you narrowed things down to the keyboard as the cause.
It sounds like the particular keyboard you are using is interrupting the boot process, and this may be happening due to the system inadvertently trying to access the USB device or boot from it. Is there anything special or unique about the keyboard you are using, for example does it have an integrated USB hub?
That the behavior changes when the keyboard is plugged into one of the front USB 3.0 ports may indicate this system is not enumerating the USB 3.0 hub in the dock before the system boots (the ports on the back of the dock are USB 2.0).
Moving forward the most straightforward option is to leave the keyboard connected to the front USB 3.0 port or switch to a different model keyboard connected to a USB 2.0 port (to help rule out the particular model keyboard as the problem) to see if that helps.
If not, the next step would be to try disabling any USB type boot options within the system BIOS to see if that helps with the behavior. While you mentioned the system is new, you may also want to check that the latest BIOS from HP is installed.
Please let us know if that information helps!