Accessibility conformance results for MUI, ahead of publication (five findings, three from one cause)

[amansoomro062]

Hello,

This is not the usual issue, so thank you for reading this far. These are accessibility conformance results from Railing, an open source project that tests React component libraries against the W3C ARIA Authoring Practices Guide and publishes the results. MUI is one of the seven libraries measured, and I wanted you to hear the results from me before anything is public.

The short version: MUI passes 59 of 64 checks. Dialog, combobox and tabs pass everything. The accordion has one finding. The menu has four, and three of those trace to a single documented decision rather than three separate defects.

Component	Result	Against
Dialog	13/13	@mui/material@9.2.0
Combobox	14/14	@mui/material@9.2.0
Tabs	13/13	@mui/material@9.2.0 + @mui/lab@9.0.0-beta.6
Accordion	10/11	@mui/material@9.2.0
Menu	9/13	@mui/material@9.2.0

How we mounted things matters here more than for most libraries, so first: components were mounted using only what MUI exports, with default configuration and no hand-written ARIA. Your documentation asks the developer to supply several attributes themselves, notably aria-haspopup, aria-controls and aria-expanded on a Menu trigger. We do not supply them, because doing so would measure how carefully we transcribed your docs rather than what MUI ships. The adapter is about fifty lines and is included in the full report below.

Three menu findings, one cause. With default mounting, the Menu trigger exposes no aria-haspopup, no aria-expanded="false" while closed, and no aria-expanded="true" while open. All three follow from the same division of responsibility your docs describe. If you regard developer-supplied ARIA as the correct contract for a Menu trigger, say so and we publish your reasoning beside the score, in full and unedited. If you would rather those defaults ship in the component, a fix inside the notice window means the fixed score is what gets published.

Two standalone findings:

1. menu.opens-on-down-arrow, serious: the menu did not open when Down Arrow was pressed on the focused trigger. The APG requires Down Arrow to open the menu and move to its first item, which is how most keyboard users reach it. Cited against the pattern and WCAG 2.1.1 Keyboard.
2. accordion.trigger-controls-panel, moderate: the accordion header control has no aria-controls referencing its panel. Cited against the pattern and WCAG 1.3.1.

These findings come with a contributor attached. Unless you would rather handle them in-house or you disagree with the findings, I intend to open PRs for the two standalone items, following your contributing guide. If the menu trigger defaults become something you want shipped in the component, count me in for that work too. A fix that ships inside the notice window means the fixed score is what gets published.

Why tell you at all? Three reasons:

1. You should hear about a public score from us rather than come across it. No score for MUI is on our index yet, and none will be for at least fourteen days. If fourteen days is not enough for your release process, ask for more and you will get it.
2. You may think we measured something wrongly, and if so, please let us know. The likeliest source of error in this project is our adapter, not your library, and we publish corrections in the open (decision log).
3. When the index goes live, these results will be on it, with a caveat we print on every page: a high score means no violations were detected by automated testing, not that a component is accessible. Your response, if you make one, is published beside the score in full.

Everything is reproducible from a clone, and every run has a replayable Playwright trace:

pnpm --filter @railing-dev/adapter-mui run dev
npx @railing-dev/runner run --target mui --component menu \
  --base-url http://localhost:5182 --repeat 3

How the measurement works: railing.dev/method · scoring · disclosure policy · source

Full report

Accessibility conformance results for MUI

You are hearing this before we publish anything. No score for MUI is on our index, and none will be for at least fourteen days, whatever this report says.

Railing runs component libraries against the W3C ARIA Authoring Practices Guide and publishes the results. Every check cites the clause it measures, every score names an exact version, and every result is run repeatedly and discarded if the answer changes.

The short version

Component	Score	Failing checks
accordion	96%	1
combobox	100%	none
dialog	100%	none
menu	77%	4
tabs	100%	none

Read this before the findings

We mounted your components like this: Mounted using only components MUI exports, with default configuration and no hand-written ARIA. MUI's documentation asks the developer to supply several ARIA attributes themselves, aria-labelledby on Dialog, and aria-haspopup / aria-controls / aria-expanded on a Menu's trigger button. Those are not supplied here, because writing them would measure our transcription of MUI's docs rather than what MUI ships. Tab panels come from @mui/lab, since @mui/material exports no tab panel component.

If that choice is the disagreement rather than the findings themselves, say so and we will publish your reasoning beside the score. It is a judgement call, not a measurement, and you are better placed to argue it than we are.

The 5 findings

5 findings. 3 of them come from 1 underlying cause, grouped below.

Three attributes your documentation asks the developer to add

3 findings, one cause. Your Menu docs show aria-haspopup, aria-controls and aria-expanded being passed to the trigger button by the developer. We do not pass them, because writing them would measure how carefully we copied your documentation rather than what MUI ships. So all three are absent together, from one decision rather than three defects. If you regard that as the correct division of responsibility, say so and we will publish your reasoning beside the score.

The menu button does not declare that it opens a menu.

• Check: menu.trigger-has-haspopup, "The menu button declares that it opens a menu"
• Component: menu
• Severity: serious, meaning the task is completable but confusing, or state is not communicated
• Why it matters: aria-haspopup is what warns a screen reader user that activating this button opens something, rather than performing an action.
• Expected: aria-haspopup="menu"
• We measured: attribute absent
• Measured against: APG pattern · WCAG 4.1.2 Name, Role, Value
• Reproduce: pnpm railing run --target mui --component menu --only menu.trigger-has-haspopup --base-url http://localhost:5182

The collapsed menu button does not expose aria-expanded="false".

• Check: menu.trigger-collapsed-by-default, "aria-expanded is present and false before the menu opens"
• Component: menu
• Severity: moderate, meaning behaviour departs from the specified pattern; users are inconvenienced
• Why it matters: Without it the user is not told the menu is currently shut, only that something can be opened.
• Expected: aria-expanded="false"
• We measured: attribute absent
• Measured against: APG pattern · WCAG 4.1.2 Name, Role, Value
• Reproduce: pnpm railing run --target mui --component menu --only menu.trigger-collapsed-by-default --base-url http://localhost:5182

The menu button does not expose aria-expanded="true" while its menu is open.

• Check: menu.expanded-state-communicated, "aria-expanded becomes true when the menu opens"
• Component: menu
• Severity: serious, meaning the task is completable but confusing, or state is not communicated
• Why it matters: A sighted user sees the menu appear. Without the state change a screen reader user is never told it did.
• Expected: aria-expanded="true"
• We measured: attribute absent
• Measured against: APG pattern · WCAG 4.1.2 Name, Role, Value
• Reproduce: pnpm railing run --target mui --component menu --only menu.expanded-state-communicated --base-url http://localhost:5182

The rest, each on its own

Not related to the above, or to each other.

The header control does not reference its panel.

• Check: accordion.trigger-controls-panel, "The header control references its panel with aria-controls"
• Component: accordion
• Severity: moderate, meaning behaviour departs from the specified pattern; users are inconvenienced
• Why it matters: The relationship is what lets assistive technology move from the control to the content it reveals.
• Expected: aria-controls referencing the panel
• We measured: attribute absent
• Measured against: APG pattern · WCAG 1.3.1 Info and Relationships
• Reproduce: pnpm railing run --target mui --component accordion --only accordion.trigger-controls-panel --base-url http://localhost:5182

The menu did not open when Down Arrow was pressed on the focused button.

• Check: menu.opens-on-down-arrow, "Down Arrow opens the menu"
• Component: menu
• Severity: serious, meaning the task is completable but confusing, or state is not communicated
• Why it matters: APG requires Down Arrow to open the menu and move to its first item, which is how most keyboard users reach it.
• Expected: menu visible
• We measured: menu absent or hidden after 2000ms
• Measured against: APG pattern · WCAG 2.1.1 Keyboard
• Reproduce: pnpm railing run --target mui --component menu --only menu.opens-on-down-arrow --base-url http://localhost:5182

How we tested it

• Versions: @emotion/react@11.14.0, @emotion/styled@11.14.1, @mui/lab@9.0.0-beta.6, @mui/material@9.2.0
• Browser: chromium 151.0.7922.34
• Configuration: default. We use only what the library exports and never hand-write ARIA, even where the documentation instructs the developer to. If we did, the score would measure how carefully we copied your docs rather than what ships in the box.
• Traces: every run above has a replayable Playwright trace, captured from exactly the run that was scored. Ask and we will send them, or regenerate them with the command below.

Reproduce any of it from a clone of the repository:

pnpm --filter @railing-dev/adapter-mui run dev
pnpm railing run --target mui --component <component> \
  --base-url http://localhost:5182 --repeat 3

Where we might be wrong

The most likely cause of a wrong result is our adapter, not your library.

It has already happened to you. Your menu scored 19% on its first run. One real defect cascaded into nine unrelated failures because our helper opened the menu with a key your trigger does not support. Corrected before this report.

So the adapter source is included below. Please tell us if we mounted your component in a way you would not recommend, or if a check misreads the specification. We will correct it and, if it has already been published, correct that too.

Where to check us

• How the measurement works, including why the runner never learns which library it is testing.
• How scoring works, with every weighting and what is deliberately left out.
• Our disclosure policy, covering notice, right of reply, and conflicts of interest.
• The decision log, which is largely a record of results we got wrong and corrected.
• The source. Every adapter, spec and score, and the code that produced this report.

What happens next

• You have fourteen days before anything about this library is published.
• If you reply, your response is published beside the score, in full and unedited.
• If you ship a fix in that window, we publish the fixed score. A finding that gets fixed before publication is the best outcome this project has, not a story we lost.
• If fourteen days is not enough for your release process, ask for more and you will get it.

The adapter we used

This is the whole of the library-specific code. There is nothing else.

accordion.tsx

import { Accordion, AccordionDetails, AccordionSummary } from "@mui/material";
import { TEXT } from "@railing-dev/harness-kit";

/**
 * MUI accordion, mounted with default configuration and all sections collapsed.
 *
 * MUI forwards props to the root element of each part, so no stamping is needed:
 * AccordionSummary's root is the control itself and AccordionDetails' root is
 * the panel.
 */
export function AccordionHarness() {
  return (
    <>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <Accordion>
        <AccordionSummary data-testid="hr-header-1">{TEXT.accordionHeader1}</AccordionSummary>
        <AccordionDetails data-testid="hr-panel-1">{TEXT.accordionPanel1}</AccordionDetails>
      </Accordion>
      <Accordion>
        <AccordionSummary data-testid="hr-header-2">{TEXT.accordionHeader2}</AccordionSummary>
        <AccordionDetails data-testid="hr-panel-2">{TEXT.accordionPanel2}</AccordionDetails>
      </Accordion>
      <Accordion>
        <AccordionSummary data-testid="hr-header-3">{TEXT.accordionHeader3}</AccordionSummary>
        <AccordionDetails data-testid="hr-panel-3">{TEXT.accordionPanel3}</AccordionDetails>
      </Accordion>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </>
  );
}

combobox.tsx

import { useEffect } from "react";
import { Autocomplete, TextField } from "@mui/material";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/** MUI Autocomplete, its combobox implementation. Default configuration. */
export function ComboboxHarness() {
  useEffect(
    () =>
      stampTestIds({
        "hr-combobox": 'input[role="combobox"]',
        "hr-listbox": '[role="listbox"]',
        "hr-option-1": { selector: '[role="option"]', index: 0 },
        "hr-option-2": { selector: '[role="option"]', index: 1 },
        "hr-option-3": { selector: '[role="option"]', index: 2 },
      }),
    [],
  );

  return (
    <>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <Autocomplete
        options={[TEXT.comboboxOption1, TEXT.comboboxOption2, TEXT.comboboxOption3]}
        sx={{ width: 300 }}
        renderInput={(params) => <TextField {...params} label={TEXT.comboboxLabel} />}
      />

      <button data-testid="hr-after" type="button">
        After
      </button>
    </>
  );
}

dialog.tsx

import { useEffect, useState } from "react";
import { Button, Dialog, DialogActions, DialogContent, DialogTitle } from "@mui/material";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * MUI dialog, mounted with default configuration.
 *
 * MUI's documentation instructs the developer to pass `aria-labelledby` to
 * Dialog themselves, pointing at the DialogTitle's id. That is not done here:
 * writing it would measure our transcription of MUI's documentation rather than
 * what MUI ships. See docs/DECISIONS.md 012.
 */
export function DialogHarness() {
  const [open, setOpen] = useState(false);
  useEffect(() => stampTestIds({ "hr-dialog": '[role="dialog"]' }), []);

  return (
    <>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <Button data-testid="hr-trigger" onClick={() => setOpen(true)}>
        {TEXT.dialogTrigger}
      </Button>

      <Dialog open={open} onClose={() => setOpen(false)}>
        <DialogTitle data-testid="hr-title">{TEXT.dialogTitle}</DialogTitle>
        <DialogContent>
          <input data-testid="hr-field-1" type="text" aria-label="Field one" />
          <input data-testid="hr-field-2" type="text" aria-label="Field two" />
        </DialogContent>
        <DialogActions>
          <Button data-testid="hr-close" onClick={() => setOpen(false)}>
            {TEXT.close}
          </Button>
        </DialogActions>
      </Dialog>

      <button data-testid="hr-outside-content" type="button">
        {TEXT.outsideContent}
      </button>
      <button data-testid="hr-after" type="button">
        After
      </button>
    </>
  );
}

menu.tsx

import { useEffect, useState } from "react";
import { Button, Menu, MenuItem } from "@mui/material";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * MUI menu, mounted with default configuration.
 *
 * MUI's documented example has the developer write aria-haspopup, aria-controls
 * and aria-expanded onto the trigger Button by hand. Those are not written
 * here. See the note in meta.ts and docs/DECISIONS.md 013.
 */
export function MenuHarness() {
  const [anchor, setAnchor] = useState<null | HTMLElement>(null);
  useEffect(
    () =>
      stampTestIds({
        "hr-menu": '[role="menu"]',
        "hr-item-1": { selector: '[role="menuitem"]', index: 0 },
        "hr-item-2": { selector: '[role="menuitem"]', index: 1 },
        "hr-item-3": { selector: '[role="menuitem"]', index: 2 },
      }),
    [],
  );

  return (
    <>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <Button data-testid="hr-trigger" onClick={(event) => setAnchor(event.currentTarget)}>
        {TEXT.menuTrigger}
      </Button>
      <Menu anchorEl={anchor} open={Boolean(anchor)} onClose={() => setAnchor(null)}>
        <MenuItem onClick={() => setAnchor(null)}>{TEXT.menuItem1}</MenuItem>
        <MenuItem onClick={() => setAnchor(null)}>{TEXT.menuItem2}</MenuItem>
        <MenuItem onClick={() => setAnchor(null)}>{TEXT.menuItem3}</MenuItem>
      </Menu>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </>
  );
}

tabs.tsx

import { useEffect, useState } from "react";
import { Tab } from "@mui/material";
import { TabContext, TabList, TabPanel } from "@mui/lab";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * MUI tabs.
 *
 * `@mui/material` exports Tabs and Tab but no tab panel, so the panel comes
 * from `@mui/lab`, which is MUI's own package, though explicitly pre-stable.
 * That is recorded in meta.ts and travels with the result: a reader deciding
 * whether to adopt MUI's tabs should know the panel half is not in the stable
 * package.
 *
 * The alternative was to hand-write the panel from MUI's documented example,
 * which would have measured our transcription rather than MUI's code.
 */
export function TabsHarness() {
  const [value, setValue] = useState("one");
  useEffect(
    () =>
      stampTestIds({
        "hr-tablist": '[role="tablist"]',
        "hr-tab-1": { selector: '[role="tab"]', index: 0 },
        "hr-tab-2": { selector: '[role="tab"]', index: 1 },
        "hr-tab-3": { selector: '[role="tab"]', index: 2 },
        "hr-panel-1": { selector: '[role="tabpanel"]', textIncludes: TEXT.panel1 },
        "hr-panel-2": { selector: '[role="tabpanel"]', textIncludes: TEXT.panel2 },
        "hr-panel-3": { selector: '[role="tabpanel"]', textIncludes: TEXT.panel3 },
      }),
    [],
  );

  return (
    <>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <TabContext value={value}>
        <TabList onChange={(_event, next: string) => setValue(next)}>
          <Tab label={TEXT.tab1} value="one" />
          <Tab label={TEXT.tab2} value="two" />
          <Tab label={TEXT.tab3} value="three" />
        </TabList>
        <TabPanel value="one">{TEXT.panel1}</TabPanel>
        <TabPanel value="two">{TEXT.panel2}</TabPanel>
        <TabPanel value="three">{TEXT.panel3}</TabPanel>
      </TabContext>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </>
  );
}

MUI carries more of the React ecosystem than any other library measured here, which is exactly why you are hearing this privately first, with time to respond however you see fit.

Aman

Search keywords:

[ZeeshanTamboli]

Thanks for your input. We are improving the accessibility of our components.

[github-actions]

This issue has been closed. If you have a similar problem but not exactly the same, please open a new issue.
Now, if you have additional information related to this issue or things that could help future readers, feel free to leave a comment.

---

@amansoomro062 Thanks again for creating this issue! If you have a moment, we would love to hear your thoughts on how we handled it with this short feedback form.