Skip to content

[Docs] "Delimiter for text" tooltip does not explain the meta-delimiter (backticks) or the parsing rule #17382

Description

@skbs-eng

Self Checks

  • I have searched for existing issues (including closed ones).
  • I confirm that I am using English to submit this report.
  • Non-English title submissions will be closed directly.
  • I have not modified this template.

Describe the problem

The "Delimiter for text" field on the dataset configuration page (and the matching "Delimiter for text" field on child chunks) accepts a single string in parser_config.delimiter / parser_config.children_delimiter, but the tooltip does not explain how that string is parsed into a list of delimiters. Users have to discover by trial-and-error that backticks are the meta-delimiter.

The current English tooltip (web/src/locales/en.ts, knowledgeDetails.delimiterTip / knowledgeDetails.childrenDelimiterTip) reads:

A delimiter or separator can consist of one or multiple special characters. If it is multiple characters, ensure they are enclosed in backticks(``). For example, if you configure your delimiters like this: \n##;, then your texts will be separated at line breaks, double hash symbols (##), and semicolons.

Problems with this wording:

  1. It never names backticks as the separator of delimiters — the user has to infer it from the example.
  2. The example is asymmetric: \n has no backticks, ## does, ; doesn't. A reader can easily read that as a typo.
  3. The example does not make clear that characters outside backticks are each treated as their own single-character delimiter (so !?; is three delimiters, not one three-character delimiter).
  4. There is no link to the backend parser, which makes the rule feel arbitrary.

This has led to repeated user confusion — see e.g. #7436 ("Multiple delimiters for Delimiter for text"), #4704 ("Delimiter setting not working"), and several related threads.

Proposed solution

Documentation-only change: rewrite knowledgeDetails.delimiterTip and knowledgeDetails.childrenDelimiterTip (and the parallel strings in other locales) so that they explicitly state the parsing rule. The rule, which is implemented in:

  • rag/nlp/__init__.pyget_delimiters()
  • deepdoc/parser/txt_parser.pyparser_txt()
  • deepdoc/parser/markdown_parser.pyMarkdownElementExtractor.get_delimiters()

is:

The field is a string. Backticks (`) group characters into a single multi-character delimiter. Any character outside backticks is itself a delimiter. Multi-character delimiters must be wrapped in backticks; single-character delimiters do not need to be. Resulting delimiters are matched longest-first.

Example: \n`##`; parses to three delimiters — newline (\n), double hash (##), semicolon (;) — and your text will be split at any of them.

A shorter second example covering only single chars (!?;!, ?, ;) would also help.

Scope

Files to change

  • web/src/locales/en.tsknowledgeDetails.delimiterTip, knowledgeDetails.childrenDelimiterTip
  • web/src/locales/<other>.ts — the matching keys in other locales, where the same ambiguity exists

Acceptance criteria

  • The tooltip names backticks as the delimiter-of-delimiters.
  • The tooltip states the single-character vs. multi-character rule.
  • The example clearly shows which characters are single-char delimiters and which are backtick-wrapped multi-char delimiters.
  • childrenDelimiterTip is updated in lockstep with delimiterTip.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions