What is your setup and OS?
I retested it now to be sure it works as expected. This is my setup:
I’m using a Gigabyte GA-Z170X-Gaming 7 motherboard. The UPD2018 is in the PCIEX1_2 slot. A Sony GDM-F520 CRT display is connected to the usbc-vga adapter. The usbc-vga adapter is connected to the USB-C port that is next to the DisplayPort input port of the UPD2018 (because the other USB-C port does not support USB-C DisplayPort alt mode). The DisplayPort input is connected to the second DisplayPort output of an Nvidia Titan X using the UPD2018’s included short DisplayPort to DisplayPort cable.
The first DisplayPort output of the Titan X is connected to a DisplayPort display because it helps to have a known working display to adjust settings for a new adapter. Or you could enable Remote Desktop or Screen Sharing or VNC.
I used a custom resolution of 2048x1536@73 Hz (329 MHz GTF timing) since 330 MHz is the max for the usbc-vga adapter.
For macOS 10.12.6, I used SwitchResX to create and select the custom resolution.
In Windows 10, I used the Nvidia control panel to create and select the custom resolution. The Device Manager shows two USB-Serial devices connected to the “ASMedia USB3.1 eXtensible Host Controller” of the UPD2018. I guess those are used for updating the firmware of the non ASM1142 chips of the UPD2018. The USB-Serial devices have a Code 10 error, but that doesn’t affect the normal functioning of the UPD2018 (DisplayPort or USB data transmission).
In Ubuntu 17.04, I used “Software Updates” to switch to Nvidia binary driver in the “Additional Drivers” tab. Then I used the “gtf” command to calculate a new custom resolution for the CRT, and the “xrandr” command to add the new mode and select it.
You must ensure that the resolution you choose has a pixel clock under 330 MHz, and is also within the horizontal (kHz) and vertical (Hz) limits of your CRT (which should be identified in the CRT’s EDID information). CRT’s usually use GTF timings. CVT is similar to GTF. An LCD may use CVT reduced blanking timings. Reduced blanking means there’s less time between the end of one frame or line and the start of the next frame or line.