decap-cms: Speed extremely slow with populating relation dropdown (several minutes)

There is a possible solution for this problem. We created an own relation widget that only uses the slug as search, display and value field. It is basically a select widget that loads the options by listing all files from a folder collection. Unfortunately, there is no function provided in the props to list all files from a directory in the git repository. I called the GitLab API directly. We have 1400+ files in the referenced collection and it works within a few seconds.

Warning: This code only works for GitLab SaaS but it can be adpted to other backends.

simple_relation/SimpleRelationControl.tsx

import React from 'react';
import { Map, List, fromJS } from 'immutable';
import { find } from 'lodash';
import Select from 'react-select';
import { reactSelectStyles } from 'netlify-cms-ui-default';
import { validations } from 'netlify-cms-lib-widgets';
import { CmsWidgetControlProps } from 'netlify-cms-core'
import { API } from 'netlify-cms-backend-gitlab'

type FileEntry = { id: string; type: string; path: string; name: string };

function optionToString(option) {
  return option && option.value ? option.value : null;
}

function convertToOption(raw) {
  if (typeof raw === 'string') {
    return { label: raw, value: raw };
  }
  return Map.isMap(raw) ? raw.toJS() : raw;
}

function getSelectedValue({ value, options, isMultiple }) {
  if (isMultiple) {
    const selectedOptions = List.isList(value) ? value.toJS() : value;

    if (!selectedOptions || !Array.isArray(selectedOptions)) {
      return null;
    }

    return selectedOptions
      .map(i => options.find(o => o.value === (i.value || i)))
      .filter(Boolean)
      .map(convertToOption);
  } else {
    return find(options, ['value', value]) || null;
  }
}

interface Props extends CmsWidgetControlProps<any> {
  t: any
  setActiveStyle(): void
  setInactiveStyle(): void
}

export default class SimpleRelationControl extends React.Component<Props> {

  state: {
    initialOptions: string[]
  }

  constructor(props: Props) {
    super(props)

    this.state = {
      initialOptions: []
    }
  }

  isValid = () => {
    const { field, value, t } = this.props;
    const min = field.get('min');
    const max = field.get('max');

    if (!field.get('multiple')) {
      return { error: false };
    }

    const error = validations.validateMinMax(
      t,
      field.get('label', field.get('name')),
      value,
      min,
      max,
    );

    return error ? { error } : { error: false };
  };

  handleChange = selectedOption => {
    const { onChange, field } = this.props;
    const isMultiple = field.get('multiple', false);
    const isEmpty = isMultiple ? !selectedOption?.length : !selectedOption;

    if (field.get('required') && isEmpty && isMultiple) {
      onChange(List());
    } else if (isEmpty) {
      onChange(null);
    } else if (isMultiple) {
      const options = selectedOption.map(optionToString);
      onChange(fromJS(options));
    } else {
      onChange(optionToString(selectedOption));
    }
  };

  async componentDidMount() {
    const { field, onChange, value } = this.props;
    if (field.get('required') && field.get('multiple')) {
      if (value && !List.isList(value)) {
        onChange(fromJS([value]));
      } else if (!value) {
        onChange(fromJS([]));
      }
    }
    const user = JSON.parse(localStorage.getItem('netlify-cms-user'))
    const directory = field.get('folder')
    if (!!user && user.backendName == 'gitlab' && !!directory && this.state.initialOptions.length == 0) {
      const api = new API({
        token: user.token,
        branch: field.get('branch'),
        repo: field.get('repo'),
        squashMerges: false,
        initialWorkflowStatus: '',
        cmsLabelPrefix: '',
        useGraphQL: false
      })
      //console.log(api)
      const response: Promise<FileEntry[]> = api.listAllFiles(directory)
      const result = await response
      //console.log(result)
      this.setState({
        initialOptions: result.map(it => it.name.replace(/\.[^/.]+$/, ""))
      })
    }
  }

  render() {
    const { field, value, forID, classNameWrapper, setActiveStyle, setInactiveStyle } = this.props;
    //const fieldOptions = field.get('options');
    const isMultiple = field.get('multiple', false);
    const isClearable = !field.get('required', true) || isMultiple;

    const options = [...this.state.initialOptions.map(convertToOption)];
    const selectedValue = getSelectedValue({
      options,
      value,
      isMultiple,
    });

    return (
      <Select
        inputId={forID}
        value={selectedValue}
        onChange={this.handleChange}
        className={classNameWrapper}
        onFocus={setActiveStyle}
        onBlur={setInactiveStyle}
        options={options}
        styles={reactSelectStyles}
        isLoading={this.state.initialOptions.length == 0}
        isMulti={isMultiple}
        isClearable={isClearable}
        placeholder=""
      />
    );
  }
}

simple_relation/schema.ts

export default {
  properties: {
    multiple: { type: 'boolean' },
    min: { type: 'integer' },
    max: { type: 'integer' },
    folder: { type: 'string' },
    branch: { type: 'string' },
    repo: { type: 'string' },
  },
  required: ['folder', 'branch', 'repo'],
};

cms.ts

import CMS from 'netlify-cms-app'
import NetlifyCmsWidgetSelect from 'netlify-cms-widget-select';
import SimpleRelationControl from './simple_relation/SimpleRelationControl';
import simpleRelationSchema from './simple_relation/schema';

// Initialize the CMS object
CMS.init()

// @ts-ignore
CMS.registerWidget('simple_relation', SimpleRelationControl, NetlifyCmsWidgetSelect.previewComponent, simpleRelationSchema)

usage in config.yml

...
fields:
- label: Source
  name: source
  widget: simple_relation
  folder: *sourceFolder
  repo: *backendRepo
  branch: *backendBranch
...

We still find this a crippling problem for editors, taking minutes (tens of minutes) for relationship fields (relation widgets, displayed as a type of custom drop-down) to populate in trying to select a post in a page. We have tried things like paring down the git repository, ensuring that editors have good amounts of memory for caching, etc.

In the most recent trials I have done as a developer who sees the issues on my machine, too, I have found that the issue is no longer requests being made from the browser on the initial use of a relationship field. In fact, I have always told the editors to give the CMS a few minutes (10-15 or more) to populate the cache after the first time they connect, so that all data can be cached properly. This seems to happen.

Once the cache is populated, however, the drop-down relation widget still take 10-15 minutes to populate (or more, depending on computer and browser) the first time one is used (remember, we use relation widgets as part of a list widget, so adding another relation to the list after one has already been added successfully works much faster). The second time a relation is added to the list and a portion of title typed, the drop-down populates within 5-10 seconds.

However, using the same template, or coming back to the same page within the same editing session, the initial population time returns to 10-15 minutes or more.

During all of this drop-down (relation widget) population time, there is almost no network traffic. None, in fact, other than a occasional “user” ping once every 5 minutes.

So this population time is all some sort of JavaScript processing, completely unrelated to Gitlab network traffic and responses.

@erezrokah

We tried 2.10.183 on an instance where we have this structure: List > Object > List > Objects with relation field. We host our own Gitlab. Each relation points to 50 to 100 files. I definitely see an improvement in speed when loading the form although there is still some loading time when revealing the last object widget and until the relations are displayed in the relation widget. It’s not snappy but it makes this form usable for us again.

Let me know if you want me to supply additional debugging information. Many thanks!

Thanks so much! We will test this in the new year and give feedback here.Am 28.12.2021 12:51 schrieb Erez Rokah @.***>: Reopened #5920.

—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

I’m going to try take a look at this in the next weeks. Thanks @erezrokah

Hello 👋 I currently have a draft PR that uses the GitLab GraphQL to retrieve 50 files at a time (to avoid hitting query complexity limit).

There are still improvements to be made (like saving to a local cache instead of in-memory cache), but it should help speed up initial load quite a bit. If you’re interested, the code is here https://github.com/netlify/netlify-cms/pull/6059/files#diff-570651114617393164ce3002abb929d59e73590014360152082e250e3075d772R468

Also, it will require some more testing to see if query complexity is impacted by the content size of files (my test repo has small files at the moment).

It would be great to get some early feedback on this, so if anyone would like to try it out at this stage, you can follow on contributing guide. See https://github.com/netlify/netlify-cms/blob/782e87c48a14937fcc7167ae5a7960a692e8054c/CONTRIBUTING.md#debugging

You’ll need to have a similar config to this:

backend:
  name: gitlab
  branch: <branch>
  repo: <repo>
  use_graphql: true

I can share the browser network traffic, but it goes on for ages - 10s of thousands of requests. I’ll try to get this for you. It’s definitely not caching.

I can provide you with a cloned repository for you to try this on, as well.

On Fri, 12 Nov 2021 at 14:52, Erez Rokah @.***> wrote:

Hi @delwin https://github.com/delwin, related to #4635 https://github.com/netlify/netlify-cms/issues/4635

There seems to be no caching of values between drop-downs despite pulling from the same source:

Initial load can be slow as it needs to load all entries from the related connection. Once entries are loaded they should be cached, so that’s definitely something we should look into. I also experience GitLab to be slower than GitHub (I don’t have official benchmarks though).

I’ll try to create a test repo to simulate this case.

Can you share the browser network traffic (by opening the browser developer tools and going to the network tab)? The browser should indicate if requests are cached or not. For example this is how Chrome shows cached requests: [image: image] https://user-images.githubusercontent.com/26760571/141477653-954681f8-5b23-42a3-9799-21e7f5e3dbf9.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netlify/netlify-cms/issues/5920#issuecomment-967133458, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC6GKEU5C5M2D4FJ5FENVDULULYZANCNFSM5GQRRPNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.