Aggressive memory optimizations#510
Draft
jaskarth wants to merge 30 commits into
Draft
Conversation
jaskarth
added
Type: Enhancement
Test Results 10 files - 10 10 suites - 10 36s ⏱️ -33s For more details on these failures, see this check. Results for commit 130f1ef. ± Comparison against base commit 218c109. ♻️ This comment has been updated with latest results. |
jaskarth
force-pushed
the
optimize/sparse-sets
branch
from
231dd1a to
29a83af
Compare
|
Maybe make this optional? |
* Start 1.12.0 * [Kotlin] Fix some generics issues * Reduce filesizes, update Kotlin metadata to 2.1.20 (#473) * Reduce filesizes, update Kotlin metadata to 2.1.20 * Fix issue with unintended additional metadata removals * [Kotlin] Fix full decompiler crash caused by missing Protobuf classes in metadata jar This was a fix from regenerating after Vineflower/kt-metadata-minifier@592068d * Fix do while loops inside of other loops being negated (#486) * Fix do while loops inside of other loops * Fix testjava11stringconcat * Fix test4 * Adjust publishing for new Central endpoint (#490) * chore(build): Adjust publishing for new Central endpoint * chore(build): Bump test-only deps, use lazy task realization in a few places * chore(build): gradle 9.0.0 * Get children methods for vararg calls (#483) * Recursively get methods through getMethodRecursive * fix test * bump ci * Improve decompiling fields (#482) * Clean up field writing to reduce redundancies * Add line numbers to field declarations when feasible * Port changes to Kotlin plugin * Revert failing tests These reverted versions fail on my machine, but Actions might think otherwise * Handle Kotlin's statements and their frequent sugar (#472) * [Kotlin] Add support for downTo, fix non-KExprent instances in other "do" statements # Conflicts: # plugins/kotlin/src/main/java/org/vineflower/kotlin/stat/KDoStatement.java # plugins/kotlin/testData/results/pkg/TestForRange.dec # plugins/kotlin/testData/results/pkg/TestFunVarargs.dec # plugins/kotlin/testData/results/pkg/TestNonInlineLambda.dec * [Kotlin] Improve for-loop resugaring * [Kotlin] Fix objects, companion object members, and whitespace * [Kotlin] Update tests * Subclass statement types instead of a custom one in its entirety * Add tests for compiling a `when` statement to switch blocks * Modify exprent replacement logic * Add accessibility to some private members * `switch` statement decompilation to `when` * Update test results and statement stuff after merge * Fix jmpWrapper calls, add `if` statement support * Update tests * Fix some instances of odd spacing in `when` blocks I should look into this more and see why it's seemingly quite inconsistent * Add test cases to `TestWhen` * Even more `when` tests * Fix additional `when` issues * Update test case * Reconfigure Kotlin variable references for more versatility in the many ways they're used * Add replace stats pass This existed prior to the rebase, but something went amiss and I'm not quite sure what * Be more lenient on allowed loops This also fixes a potential issue where `repeat` would be generated despite counting down * Fix plugins not loading when a ZipFS has already been opened for the current Jar (#475) * Add flag to detect wrong variable merges (#477) * Add flag to detect wrong variable merges * Copy dec file from ci * Add checks for generic types and discard any unexpected types (#430) * Add corrupted signature test * Add mismatched generics catches * Fix all the cases where the tests indicated I messed up Also upload the intended test result for TestCorruptedSignatures * Delegate semicolons to plugins when possible (#471) * Delegate statement semicolon appending to plugins when possible * Update Kotlin tests (now without semicolons) * Add cautionary tale * Compact canonical constructor support (#484) * Add support for compact constructors * Support private constructors * Cleanup * comment * Rename method * Remove public + static from class definitions in interfaces (#491) * Remove public + static from class definitions in interfaces * Fix tests * Fix build * Improve annotation formatting (#497) * Fix desync of exception tracking (#494) * Be more forgiving when parsing arguments for paths (#493) Instead of counting the non-options and using that to determine the last file path for the destination, instead store the last file path and process that lazily. This accounts for the cases where a single 'pseudo-options' (--only, --silent, --add-external) is specified alongside a single file path, where it previously incorrectly determined the file path to be the destination. This also accounts for the case where pseudo-options are specified at the end of the command-line invocation, rather than before all file paths. Fixes #492 * Add new test * Properly fix bug that led switch on string to produce null with tableswitches * Remove field debug, fix lambda CME, and try to improve nested lambda var handling * Initial if statement prettifier based on heuristics * Attempt to appease Github Actions * Fix generic erasure for nested generics and add test cases (#503) * Add type annotations test, always run gradle tests * Implement type annotations * Fix writing array types with type annotations * Fix generic erasure for lambdas (#504) * Account members for enum + hide private for enum constructors (#502) * Account members for enum, + hide private for enum constructors * Revert test * Fix terrible merge * Fix exception variables losing their LVT * Make stack processing keep track of the lvts that it replaces * Add nodebug lvt test, fix some NPEs * Cleanup, document, and improve variable processing * Refactor the way bytecode offsets are tracked. * Split SimpleInstructionSequence and FullInstructionSequence * Remove usages of VBStyleCollection * Instructions now track their own bytecode offsets. * Don't inline returns in switch expressions * Undo test change * Rename illegal methods at callsites, don't write illegal variable names * Add comma separation support to -e and --only, improve class allowlisting * Fix improper elision of varargs, always cast nulls in varargs * Fix wrong method being called for overloads with arrays * Cleanup bugs in previous patches * Fix IOOBE * Fix string concat with 1 element not being recognized * Don't crash when encountering condys in indys * Properly propagate upper bounds through ternaries * Fix int constants in ternaries becoming chars, fix cases of lower > upper bound in type analysis * Make assertion into validation again * Modify Kotlin flag handling (#474) * Convert to using metadata's Flags util class universally Modified metadata jar made in Vineflower/kt-metadata-minifier@5bd1751 * Fix incorrect `inline` treatments Also whoops classes never had `inline` (they've had `value` this whole time) * Update metadata to 2.2.20 * Fix a bunch of incorrect variables (#507) * add catchtempvar * Fix synchronized and foreach * Fixes for record pattern matching and fix for synchronized block * Fix switch on string * Fix try with resources that only reference a variable * Some more fixes * Update tests * Update tests again * Update TestLVTReassignmentNoDebug * Fix TestDoubleForEach * Fix switch pattern matching * Fix switch tests * Fix TestSwitchPatternMatching21 * Fix TestSwitchPatternMatching6 * Update kotlin tests * Add some comments and improve synthetic variable detection on enum switches * Add comments * Add monitor bytecode offsets and comment about broken test * Fix multi exception catches losing their type, write them in the right order * Fix writing text token class names * Fix errors in Kotlin's decompilation caused by 31d9e1c (#511) * Close copyEntry output streams, reorganize tests * Fix LVT names being used when LVT names are disabled * Add unknown variable test * Fix parsing error with switches inside a finally * Fix tests * Fix many validation bugs * Unify and optimize try with resources processing * Fix errors from decompiling Kotlin classes (#513) * Fix the generics checker crashing on mismatched generic counts * Don't handle bytecode offsets in KProperty they're already handled by KotlinWriter whoops * Try to fix generics erasure from synthetic parameters * Allow for mismatches on the assumption it won't crash and burn as a result * Unify try merging, fix stat/expr toString changing imports * Remove getSequentialObjects * Fix infinite loop in generic arg mapping * Fix array type assertion in invocation type inference * Fix NPE in generics processing * Test `lcmp`, `fcmp` and `dcmp` to `boolean` in TestNumberCompareToBoolean (#518) * test: Test `lcmp`, `fcmp` and `dcmp` to `boolean` in TestNumberCompareToBoolean All the tests are functionally `number == number && number != 0`, baking in a NaN check. For the extra curious, the NaN checks in the floating point tests work by taking advantage of the same fcmp insn will always send NaN to 1 or -1, allowing for xor to collapse it to 0. `iconst_1`, `iand` are evidently redundant but can be helpful if you need to strictly fit in one bit. * test: Add commented decompiled sequences to TestNumberCompareToBoolean * Add more unit tests * Prefix counter to exported dot files, so they appear in chronological order (#520) * Prefix counter to exported dot files, so they appear in chronological order * Use CounterContainer * Use remapped Kotlin metadata * Aggressively optimize Kotlin metadata size * Show try catch blocks in bytecode dump * Add Java 25 tests and PPMM tests * Include Java Runtime classes by default * Use the right index for writing type annotations * Add --method-to-decompile option (#519) * Move metadata out of global state (#515) * De-globalize Kotlin metadata * Add missing null checks * Move wrapper requirements to support earlier extra parsing * Pull higher-level objects into Kotlin metadata attribute * Refactor struct attributes, optimize attribute construction * Add more record tests, move some processor packages * Fix patterns with primitives and nested patterns in a switch (#530) * Fix patterns with primitives and nested patterns in a switch * Remove debug print * Add a list for StatEdges to remove * Remove synthetic Objects.requireNonNull from constructors * Add dark theme for dot exports (#526) * Fix primitives in pattern with an if statement with `false` (#531) * Fix primitives in patterns when the if statement has `false` * Add null check * Add test * Add tests for java 25 method references * Add more method reference tests * Remove ARM checks to fix old Java tests Foojay Disco API supports aarch64 builds of Java 9 and 10 for Linux, and Gradle supports x86_64 builds on macOS * Add support for markdown javadoc (#533) * Add helper case functions (#545) * feat: Add helper case functions Signed-off-by: aoqia <aoqia@aoqia.dev> (cherry picked from commit decdb6e) * docs: Add method docstrings for case helper methods Signed-off-by: aoqia <aoqia@aoqia.dev> * refactor: Use variable instead of redundant check Signed-off-by: aoqia <aoqia@aoqia.dev> * fix: Place default case at start instead of end Signed-off-by: aoqia <aoqia@aoqia.dev> * style: Remove accidental whitespace Signed-off-by: aoqia <aoqia@aoqia.dev> * style: Remove unnecessary annotations Signed-off-by: aoqia <aoqia@aoqia.dev> * style: Make method a little more readable Signed-off-by: aoqia <aoqia@aoqia.dev> * refactor: Remove uses of final and var Signed-off-by: aoqia <aoqia@aoqia.dev> --------- Signed-off-by: aoqia <aoqia@aoqia.dev> * Improve synthetic non-null checks generated by modern javac * Cleanup redundant requireNonNull checks * Inline synthetic vars before method reference lambdas Co-Authored-By: coehlrich <coehlrich@users.noreply.github.com> * Remove dead code * Using pattern matching (#546) * Using pattern matching * Remove extra indentation * Remove extra indentation and further cleanup * Kotlin: decompile most lambdas and anonymous objects (#525) * De-globalize Kotlin metadata * Add missing null checks * Move wrapper requirements to support earlier extra parsing * Pull higher-level objects into Kotlin metadata attribute * Initial work on Kotlin lambdas * Handle lambda classes, mostly * Fix zero-parameter formatting * Fix post-merge issues * Handle lambdas a bit better * Improve exit detection * Function reference support * Anonymous object hack * Fix wrong parameter names, add ignored parameters test * Revert removing some requireNonNull checks * Properly implement assert decompilation - Fixed asserts in interface default methods not decompiling - Fixed assignment asserts not decompiling properly - Removed empty space between asserts - Fixed asserts being pulled into loops - Fixed if statements with a single assert from creating an invalid assert - Added support for marking improper assert decompilation * Fix invalid ppmm creation with floats and doubles * Simplify break structure for string switches (#544) * feat: Simplify break structure for string switches As a reference, this is found in TestEclipseSwitchString. Signed-off-by: aoqia <aoqia@aoqia.dev> (cherry picked from commit cedf703) * style: Use 2 spaces indent Signed-off-by: aoqia <aoqia@aoqia.dev> * feat: Improve exit helper break structure Signed-off-by: aoqia <aoqia@aoqia.dev> * fix: Prevent if statements without a basichead from being processed Signed-off-by: aoqia <aoqia@aoqia.dev> * style: Remove left-in note tag Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * refactor: Change variable names and comments for clarity Signed-off-by: aoqia <aoqia@aoqia.dev> * docs(code): Clean up some code comments Signed-off-by: aoqia <aoqia@aoqia.dev> * style: Invert if guard for parity with rest of func Signed-off-by: aoqia <aoqia@aoqia.dev> * chore: Add null check and nullable Signed-off-by: aoqia <aoqia@aoqia.dev> --------- Signed-off-by: aoqia <aoqia@aoqia.dev> Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * Fix pattern match asserts from dropping code * Fix NPE in assert processing * Add include classes option and external test * Fix missing successor in ExitHelper * Fix exprent list NPE in ExitHelper * Fix infinite looping during generics checking (#549) * Fix crashes when encountering CONSTANT_MethodType * Fix incorrectly renamed variables (#551) * Fix incorrectly renamed variables * Fix switches and tests * Fix test * Handle object construction and related constructs (#550) * Handle object construction (in theory) universally * Fix array creation * Fix array creation in annotation class declarations # Conflicts: # plugins/kotlin/src/main/java/org/vineflower/kotlin/KotlinWriter.java * Fix foreach loop creation when iterator is used afterwards * Fix unit test * Fix finally processing in plugins (#554) * Fix missing if successor after loop elimination * Fix regression with patch * Kotlin metadata rework, part 2 (#552) * Metadata rework p2: metadata now lives with the objects they represent! this is largely for allowing invocation exprents and such resugar into their Kotlin counterparts much more easily (less guessing) * Hide EnumEntries synthetic field * Fix issues stemming from companion object fields * Fix specific oddities relating to property backing field identification * Fix crashes from unsigned constants * Improve plugin support and hooks (#523) * Changes to plugins * More plugin work, add root processor support * Improve class pass api * Fix merge * Fix unit tests * [Kotlin] Fix regressions from metadata rework * Kotlin properties and infix functions (#556) * Property access test * Add property and infix fun support * Fix crashes from static method calls * Fix precedence for properties * writing code is hard * Missed a test? * Decompile Kotlin's JVM multifile classes (#557) * Multi-file decompilation * Add multifile test * Add tests for finally vars * Recognize iinc in finally block * Fix NPE in Kotlin annotation processing * Add test for switch pattern matching with string * Improve heuristic for creating for loops * Fix NPE in mapGenVarsTo * Fix empty super() not removed when constructor has label * Fix stack vars processing unable to inline into foreach loops * Fix incorrectly renaming pattern match variables (#555) * Fix incorrectly renaming pattern match variables * Add test and indent switch * Avoid pruning most user written variables with no use * Identify more ternaries and ppmm in Kotlin, add newline for labeled ifs * Don't inline variables in StackVars if the target is ppmm * Fix NPE when checking generics * Reduce the amount of exported dots * Rewrite string-switch processing (#558) * chore: Move changes over from feat/string-switch-rewrite * fix: Dup var not being found sometimes * chore: Generate test dec * chore: Add test dec changes * refactor: Remove now-useless code * fix: Default case edge not being added with addCase * style: Remove noinspection IDEA comment Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * refactor: Remove unnecessary block * style: Remove comment Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * style: Remove extra whitespace Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * style: Remove accidental formatting * chore: Revert implementation of VBCollection#remove * refactor: Remove all usage of Optional * style: Remove name and commit hash in comment fix: Add an extra null check just in case * fix: Use removeWithKey instead of deprecated remove * refactor: Use removeCase helper function --------- Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * Add Automatic-Module-Name to the manifest (#559) * add Automatic-Module-Name to both slim and all jars * Change the module name to match fernflower * Fix not being able to find the while true loop for switches with a (#561) return * Inline returns in try with finally (#562) * Inline returns for try with resources * Inline returns in finally * Remove unused label * Add return positioning test for try with resources and add support for inline returns when the try with resources has multiple resources * Update tests * Revert "Update tests" This reverts commit 194f89d. * Revert "Add return positioning test for try with resources and add support for" This reverts commit d1dcace. * Revert "Inline returns for try with resources" This reverts commit d39ce60. * Fix switch expressions not being inlined when none of the case statements directly contain `yield` (#563) * Fix switch expressions not being inlined when none of the case statements directly contain `yield` * Removed old loop * Update TestUnknownCastJ17 * Add plugin hook just before decompilation * Allow for introspection of units * Implement return inlining for try-with-resources Co-authored-by: jaidaken <jamiehewitt@pm.me> Co-authored-by: coehlrich <coehlrich@users.noreply.github.com> * Add PR templates and clarify AI policy * Fix index out of bounds with switch-on-string with null * Fix NPE in clashing var remapping * Fix NPE in kotlin for each loop processing * Fix Kotlin NPE with companion classes --------- Signed-off-by: aoqia <aoqia@aoqia.dev> Co-authored-by: sschr15 <steven.sschr15@hotmail.com> Co-authored-by: coehlrich <coehlrich@users.noreply.github.com> Co-authored-by: zml <zml@stellardrift.ca> Co-authored-by: Owen <23108066+Owen1212055@users.noreply.github.com> Co-authored-by: shartte <shartte@users.noreply.github.com> Co-authored-by: Kroppeb <7889478+Kroppeb@users.noreply.github.com> Co-authored-by: Joseph Burton <burtonjae@hotmail.co.uk> Co-authored-by: sciwhiz12 <sciwhiz12@gmail.com> Co-authored-by: Ampflower <ampflower@outlook.com> Co-authored-by: aoqia194 <30682549+aoqia194@users.noreply.github.com> Co-authored-by: aoqia <aoqia@aoqia.dev> Co-authored-by: TropheusJ <60247969+TropheusJ@users.noreply.github.com> Co-authored-by: jaidaken <jamiehewitt@pm.me>
* Reuse existing logic to fix wrong ppmm inline match Fixes #569 * Update dec file * Improve pmm inlining more
* Add debug marker exceptions * Add debug marker exceptions * Apply suggestions from code review Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * Move marker exceptions into dedicated source set * Update comment --------- Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com>
* Fix 2 bugs in multi entry exception ranges * Update dec file * Feature/better multi entry exception deobf (#586) * Improve output for some obfuscated exceptions * Update dec file * A tad more docs * Update src/org/jetbrains/java/decompiler/modules/code/ExceptionDeobfuscator.java Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> * Reword comment * update dec file --------- Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com> --------- Co-authored-by: Jasmine Karthikeyan <25208576+jaskarth@users.noreply.github.com>
* temp commit to trigger diff machine * Fix mistake * Strict null * More cleanup and docs * More cleanup * I think I went too far * fix wrong cleanup * Fix typo's etc in the docs * comments * Don't use strings
* Add missing instruction offsets * Revert of MinecraftForge/ForgeFlower/pull/105 Revert of d88f743 Fixes #596 * temp undo * Fix dec file issues * Update kotlin dec file
jaskarth
force-pushed
the
optimize/sparse-sets
branch
from
2a87a7e to
0ff02f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change sparse sets, sforms maps, and exprent bytecode offsets to prefer non-array backing implementations to prevent extremely sparse arrays and reduce peak memory usage when decompiling very large methods.