Tech Debt: Duplicate Avro/POJO Model Generation in models and models-pojo

[Wisienkas]

Tech Debt: Duplicate Avro/POJO Model Generation in models and models-pojo

Summary

The sdks/models and sdks/models-pojo modules serve overlapping purposes and create confusion and correctness issues when used incorrectly. This issue tracks an investigation and cleanup of the two modules, with a proposed path to consolidation.

Background

The project currently maintains two parallel model generation pipelines for the same Avro schemas:

• sdks/models — generates proper Avro SpecificRecord classes via avro-maven-plugin. Classes have getClassSchema(), are binary-compatible with all Avro readers/writers, and produce correct Avro binary encoding. The .avsc files live in src/main/avro/ but are not packaged into the jar as classpath resources.
• sdks/models-pojo — generates Lombok POJOs from the same schemas. No getClassSchema(), no embedded schema. Compatible with Spark Encoders.bean() (which requires standard Java bean conventions), but produces incorrect Avro binary encoding when written via spark.write().format("avro") without an explicit schema option — Spark infers the schema from reflection, causing all nullable String fields to be union-wrapped as {"string": "value"} in the binary output.

This ambiguity has caused real bugs: the DwcDpVerbatimConverter initially imported from models-pojo, which meant the output Avro was union-wrapped and not readable by downstream pipeline steps. The workaround was to copy extended-record.avsc into the Spark module resources and pass .option("avroSchema", ...) explicitly on write — a fragile solution that shouldn't be necessary long-term.

Concrete encoding difference

The POJO/Lombok generated classes in models-pojo don't expose the Avro schema, causing Spark to infer it from reflection. This results in all String fields being treated as ["null", "string"] union types, and map fields gaining an extra map wrapper element. The incorrect output looks like:

{
  "coreId": { "string": "e4d2f0a0-64f8-4f23-9569-d2e005839f73" },
  "coreRowType": { "string": "http://rs.tdwg.org/dwc/terms/Event" },
  "coreTerms": {
    "map": {
      "http://rs.tdwg.org/dwc/terms/day": { "string": "12" },
      "http://rs.tdwg.org/dwc/terms/eventID": { "string": "e4d2f0a0-64f8-4f23-9569-d2e005839f73" },
      "eventCategory": { "string": "Event" },
      "http://rs.tdwg.org/dwc/terms/country": { "string": "México" }
    }
  }
}

When the schema is supplied explicitly (via models or .option("avroSchema", ...)), the output is correctly encoded as:

{
  "coreId": "e4d2f0a0-64f8-4f23-9569-d2e005839f73",
  "coreRowType": "http://rs.tdwg.org/dwc/terms/Event",
  "coreTerms": {
    "http://rs.tdwg.org/dwc/terms/day": "12",
    "http://rs.tdwg.org/dwc/terms/eventID": "e4d2f0a0-64f8-4f23-9569-d2e005839f73",
    "eventCategory": "Event",
    "http://rs.tdwg.org/dwc/terms/country": "México"
  }
}

The union-wrapping of scalar fields and the spurious map wrapper around coreTerms both cause downstream pipeline steps to fail to read the Avro output.

Investigation Required

Before proposing a fix, the following should be established:

• Which schemas are still actively used? The hypothesis is that only ExtendedRecord (and possibly a small number of others) are still used in practice. A codebase scan across gbif-pipelines and related repos should confirm this.
• Which modules depend on models vs models-pojo? Map out who imports which, and whether any consumer actually requires the Lombok POJO behaviour (e.g. Encoders.bean() usage).
• Are the .avsc files identical across both modules? Confirm whether the schemas have diverged.

Proposed Fix Options

Once the investigation is complete, the likely path forward is one of:

1. Collapse into models only — If models-pojo is only needed for Encoders.bean() compatibility in Spark jobs, switch those jobs to use Dataset<Row> / schema-explicit writes instead, removing the need for POJO encoding entirely. Embed the .avsc files as classpath resources in models so getClassSchema() works without copying schema files into consumers.

2. Permanently inline the generated POJOs — If the Lombok POJOs are genuinely useful and the generation is fragile, replace generated output with hand-maintained (or once-generated, then committed) source files, removing the generation step.

3. Hybrid: keep models for Avro I/O, extend it for Spark compatibility — Adjust models so the generated classes are usable with Spark (e.g. by registering Kryo serializers or using schema-explicit Dataset writes), deprecating models-pojo entirely.

Acceptance Criteria

• Codebase scan completed: list of active schemas and which modules consume models vs models-pojo
• Proposal for consolidation agreed with the team
• models-pojo either removed or clearly scoped with a documented rationale for its existence
• No Spark job requires copying .avsc files locally as a workaround for missing classpath resources
• Downstream pipeline steps can read Avro output without schema mismatch