Many of the AI generated ERT tests that were recently added make intrusive assumptions about the implementations of the functions being tested, such that refactoring becomes difficult. Here is an example:
(ert-deftest ledger-report/change-month ()
;; Bind ledger-report-ledger-buf to a real buffer for `expand-format-specifiers'.
(let ((report-buf (get-buffer-create ledger-report-buffer-name))
(src-buf (current-buffer)))
(unwind-protect
(let ((ledger-report-current-month '(2024 . 5))
(ledger-report-name "X")
(ledger-report-ledger-buf src-buf)
(ledger-reports '(("X" "ledger reg --period %(month)")))
(ledger-report-format-specifiers
'(("month" . ledger-report-month-format-specifier))))
(cl-letf (((symbol-function 'ledger-report-redo) (lambda (&rest _) nil))
((symbol-function 'ledger-reports-custom-save)
(lambda () nil)))
(ledger-report--change-month -1)
(should (equal '(2024 . 4) ledger-report-current-month))
(ledger-report--change-month 5)
(should (equal '(2024 . 9) ledger-report-current-month))))
(let ((kill-buffer-query-functions nil)) (kill-buffer report-buf)))))This test creates a report buffer but without actually initializing it in ledger-report-mode, and sets specific buffer-locals based on implementation details. It also stubs out a few functions that happen to be called in the current implementation of change-month but which aren't, in a change I'm trying to make. Rather than simply creating a real report buffer based on the user-facing API and examining its contents as various commands are called, this test's assertions tends to freeze the existing implementation and its internals in place.
I think it would be worth auditing all of the newly added tests and rewriting them to use helpers like ledger-tests-with-temp-file in combination with executing the interactive commands that form ledger-mode's public interface. User input can be simulated with ledger-tests-with-simulated-input instead of stubbing out various ledger-read-* helpers.
Many of the AI generated ERT tests that were recently added make intrusive assumptions about the implementations of the functions being tested, such that refactoring becomes difficult. Here is an example:
This test creates a report buffer but without actually initializing it in
ledger-report-mode, and sets specific buffer-locals based on implementation details. It also stubs out a few functions that happen to be called in the current implementation ofchange-monthbut which aren't, in a change I'm trying to make. Rather than simply creating a real report buffer based on the user-facing API and examining its contents as various commands are called, this test's assertions tends to freeze the existing implementation and its internals in place.I think it would be worth auditing all of the newly added tests and rewriting them to use helpers like
ledger-tests-with-temp-filein combination with executing the interactive commands that form ledger-mode's public interface. User input can be simulated withledger-tests-with-simulated-inputinstead of stubbing out various ledger-read-* helpers.