Yeah, it's proportional to the total number of GDI objects. Approximately:

  DWORD HmgNextOwned(DWORD index, DWORD pid, HANDLE * handle) {
    GDI_TABLE_ENTRY entry;
    GreAcquireHmgrSemaphore();
    // GetNextEntryIndex is inlined in reality, this is the main loop you see in the disassembly
    while(index = gpHandleManager->GetNextEntryIndex(index, &entry)) { 
      if(entry.Type != 0 && ((pid & 0xFFFFFFFD) ^ entry.ProcessId) & 0xFFFFFFFE) == 0) {
        *handle = index | (entry.Upper << 16);
        break;
      }
    }
    GreReleaseHmgrSemaphore();
    return index;
  }

Best information I could find on the relevant structures is [1], which gives an idea of what you're traversing. I don't know why this is designed this way; it does seem suboptimal.

[1] https://cvr-data.blogspot.com/2016/11/windows-10-anniversary...