Skip to content

Commit 8cf4678

Browse files
DanRoscignoDanRoscigno
committed
testing
Signed-off-by: DanRoscigno <dan@roscigno.com>
1 parent e32ddf5 commit 8cf4678

4 files changed

Lines changed: 624 additions & 5 deletions

File tree

‎README.md‎

Lines changed: 60 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -429,18 +429,74 @@ The tool provides clear error messages for common issues:
429429
- Network connectivity issues
430430
- API rate limiting
431431

432+
## Testing
433+
434+
The `examples/` directory contains a test corpus and an automated checker.
435+
436+
### Test corpus: `examples/StarRocksTest.md`
437+
438+
A curated set of patterns drawn from real StarRocks documentation that have caused translation problems in the past:
439+
440+
| Pattern | Why it matters |
441+
|---------|---------------|
442+
| YAML frontmatter | Must be preserved exactly |
443+
| HTML in Markdown table cells (`<ul><li>`, `<br />`, `<code class="...">`) | Tags must not be translated or restructured |
444+
| Tilde fence code blocks (`~~~SQL`) | Must be converted to backtick fences cleanly |
445+
| MDX `import` statements and `<Tabs>`/`<TabItem>` JSX | Must be preserved unchanged |
446+
| Template variables in code (`{{ data_interval_start }}`) | Airflow/dbt syntax must not be touched |
447+
| HTML comparison tables with `colspan` | Full HTML blocks must pass through untranslated |
448+
| Admonitions indented inside numbered lists | Indentation must survive translation |
449+
| `<details>` collapsible blocks | Content indentation must be preserved |
450+
| Cross-references with relative paths and anchors | Only the display text is translated; the URL is not |
451+
452+
### Automated checker: `examples/check_translation.js`
453+
454+
After translation, the checker runs 13 static checks against the source/output pair and reports PASS/FAIL for each:
455+
456+
- No `__MTX_` placeholder leaks
457+
- Heading count
458+
- Code block count and non-comment content
459+
- Link URL preservation
460+
- HTML tags in table cells
461+
- Frontmatter preserved exactly
462+
- Import statements preserved
463+
- Admonition marker count
464+
- Admonition indentation (catches the "indented :::note gets unindented" bug)
465+
- Never-translate term spot-check
466+
- Unordered list item count
467+
- Table column counts
468+
469+
### npm scripts
470+
471+
```bash
472+
npm test # Translate StarRocksTest.md → zh, then run all checks
473+
npm run test:ja # Translate StarRocksTest.md → ja, then run all checks
474+
npm run check:zh # Re-run checks on an already-translated StarRocksTest_zh.md
475+
npm run check:ja # Re-run checks on an already-translated StarRocksTest_ja.md
476+
```
477+
478+
`check:zh` and `check:ja` are useful for iterating on the system prompt or dictionaries without calling the API again.
479+
432480
## Development
433481

434482
### Project Structure
435483

436484
```
437485
doc-translator/
438486
├── bin/
439-
│ └── cli.js # CLI entry point
487+
│ └── cli.js # CLI entry point
440488
├── src/
441-
│ └── translator.js # Core translation logic
442-
├── package.json # Dependencies and scripts
443-
└── README.md # Documentation
489+
│ ├── translator.js # Base class and shared utilities
490+
│ ├── translator_ast_mvp.js # AST-based translator (default)
491+
│ └── configs/
492+
│ ├── system_prompt.txt # Translation instructions for the model
493+
│ ├── never_translate.yaml # Terms that must never be translated
494+
│ └── language_dicts/ # Per-language translation dictionaries
495+
├── examples/
496+
│ ├── StarRocksTest.md # Test corpus
497+
│ └── check_translation.js # Automated output checker
498+
├── package.json
499+
└── README.md
444500
```
445501

446502
### Architecture

‎examples/StarRocksTest.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
---
2+
displayed_sidebar: docs
3+
toc_max_heading_level: 4
4+
keywords: ['StarRocks', 'translation test', 'Iceberg', 'Hive']
5+
---
6+
17
# Sample Markdown Document
28

39
If you define configuration items in the custom catalog and want configuration items to take effect when you query data, you can add the configuration items to the `PROPERTIES` parameter as key-value pairs when you create an external table. For example, if you define a configuration item `custom-catalog.properties` in the custom catalog, you can run the following command to create an external table.
@@ -339,3 +345,159 @@ insert into external_t select * from other_table;
339345
- [ALTER TABLE](ALTER_TABLE.md)
340346
- [DROP TABLE](DROP_TABLE.md)
341347

348+
## HTML in Table Cells
349+
350+
Some StarRocks docs use HTML inside Markdown table cells to support complex multi-item content. The HTML tags must be preserved exactly:
351+
352+
| External Data Source | Supported Scenarios | Stable Versions |
353+
| :------------------- | :------------------ | :-------------- |
354+
| Hive | <ul><li>Non-partitioned table: v2.5.4 & v3.0+</li><li>DATE and DATETIME-type partition: v2.5.4 & v3.0+</li><li>STRING-type Partition Key to DATE: v3.1.4+</li></ul> | v2.5.13+<br />v3.0.6+<br />v3.1.5+ |
355+
| Iceberg | <ul><li>Non-partitioned table: v3.0+</li><li>Partition Transform: v3.2.3</li><li>Partition-level refresh: v3.1.7 & v3.2.3</li></ul> | v3.1.5+<br />v3.2+ |
356+
357+
Cells may also contain inline HTML: set the `index` parameter to <code class="language-text">hello*</code> to retrieve all indexes whose names start with `hello`.
358+
359+
## Tilde Fence Code Blocks
360+
361+
Some older StarRocks docs use tilde fences (~~~) instead of backtick fences. These must be processed correctly:
362+
363+
~~~SQL
364+
CREATE TABLE tbl (k1 int, v1 int sum)
365+
DISTRIBUTED BY HASH(k1)
366+
BUCKETS 8
367+
PROPERTIES(
368+
"colocate_with" = "group1"
369+
);
370+
~~~
371+
372+
~~~Plain Text
373+
SHOW PROC '/colocation_group';
374+
~~~
375+
376+
## MDX Imports and JSX Components
377+
378+
StarRocks MDX files use Docusaurus Tabs components for multi-platform documentation:
379+
380+
import Tabs from '@theme/Tabs';
381+
import TabItem from '@theme/TabItem';
382+
383+
<Tabs groupId="storage">
384+
<TabItem value="AWS" label="AWS S3" default>
385+
386+
Configure your StarRocks cluster to access AWS S3 storage. Choose one of the following authentication methods:
387+
388+
- Instance profile (recommended for production)
389+
- Assumed role
390+
- IAM user
391+
392+
</TabItem>
393+
394+
<TabItem value="HDFS" label="HDFS">
395+
396+
Configure your StarRocks cluster to access HDFS storage.
397+
398+
:::tip
399+
400+
If an error indicating an unknown host is returned when you send a query, add the mapping between host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** file.
401+
402+
:::
403+
404+
</TabItem>
405+
</Tabs>
406+
407+
## Template Variables in Code
408+
409+
Integration docs for Airflow and dbt use double-brace template syntax inside code blocks. These must not be translated:
410+
411+
```sql
412+
-- Load new rows since the last run
413+
SELECT *
414+
FROM source_table
415+
WHERE loaded_at >= '{{ data_interval_start }}'
416+
AND loaded_at < '{{ data_interval_end }}'
417+
AND record_id NOT IN (SELECT record_id FROM target_table)
418+
```
419+
420+
The dbt `config` block uses the same double-brace syntax:
421+
422+
```sql
423+
{{ config(
424+
materialized='table',
425+
indexes=[{"columns":["order_id"]}]
426+
)}}
427+
428+
SELECT * FROM {{ source('your_source', 'orders') }}
429+
JOIN {{ source('your_source', 'users') }} USING (user_id)
430+
```
431+
432+
## HTML Comparison Table
433+
434+
Data lake documentation uses full HTML tables with colspan for comparison grids:
435+
436+
<table>
437+
<thead>
438+
<tr>
439+
<th>&nbsp;</th>
440+
<th>Data Cache</th>
441+
<th>Materialized view</th>
442+
<th>Native table</th>
443+
</tr>
444+
</thead>
445+
<tbody>
446+
<tr>
447+
<td><b>Data loading and updates</b></td>
448+
<td>Queries automatically trigger data caching.</td>
449+
<td>Refresh tasks are triggered automatically or manually.</td>
450+
<td>Supports various import methods but requires manual maintenance.</td>
451+
</tr>
452+
<tr>
453+
<td><b>Query performance</b></td>
454+
<td colspan="3">Data Cache &le; Materialized view = Native table</td>
455+
</tr>
456+
</tbody>
457+
</table>
458+
459+
## Admonition Inside a Numbered List
460+
461+
When an admonition appears inside a numbered list item, its indentation must be preserved:
462+
463+
1. Create the database and schema in StarRocks.
464+
465+
2. Load your data into the staging table.
466+
467+
:::note
468+
Ensure that the staging table schema matches the target table schema. If the schemas differ, the load job will fail with a schema mismatch error.
469+
:::
470+
471+
3. Insert data from the staging table into the target table.
472+
473+
:::caution
474+
Running INSERT OVERWRITE replaces all existing data in the target partition. Verify your filter conditions before executing.
475+
:::
476+
477+
4. Verify the row counts match between staging and target.
478+
479+
## Details Block
480+
481+
The HTML `<details>` element creates a collapsible section. Content indentation must be preserved:
482+
483+
<details>
484+
<summary>Advanced configuration options</summary>
485+
486+
- `max_scan_key_num`: Maximum number of scan keys evaluated per query (default: 1024).
487+
- `enable_profile`: Enable query profile collection for performance analysis (default: false).
488+
- `query_timeout`: Maximum query execution time in seconds before cancellation (default: 300).
489+
490+
</details>
491+
492+
## Cross-References with Anchors
493+
494+
Links with relative paths and in-page anchors must have their URL portion preserved unchanged:
495+
496+
For more information, see the following resources:
497+
498+
- [Query Planning](../best_practices/query_tuning/query_planning.md)
499+
- [JOIN Operations](../sql-reference/sql-statements/table_bucket_part_index/SELECT/SELECT.md#join)
500+
- [ALTER RESOURCE](../sql-reference/sql-statements/Resource/ALTER_RESOURCE.md)
501+
- [Data type mapping](External_table.md#Data-type-mapping)
502+
- [Iceberg catalog](./catalog/iceberg/iceberg_catalog.md)
503+

0 commit comments

Comments
 (0)