Improve CRLF support in lexer - #1414
Conversation
|
effekt/effekt/shared/src/main/scala/effekt/Lexer.scala Lines 330 to 335 in e15eca1 is likely also a source of inconsistencies (if it doesn't "magically" work out somehow) |
| private def shebang(): TokenKind = | ||
| advanceWhile { (curr, _) => curr != '\n' } | ||
| advanceWhile { (curr, _) => curr != '\n' && curr != '\r' } | ||
| val command = getCurrentSlice(skipAfterStart = 2) // Remove `#!` | ||
| TokenKind.Shebang(command) | ||
| } |
There was a problem hiding this comment.
Though I don't expect too many CRLF enjoyers using shebangs, we should have a consistent "until the end of the line" predicate, hence this funny-looking change.
|
And there are still some tests failing that probably should work, like |
This comment was marked as outdated.
This comment was marked as outdated.
|
@martin-ilgner if you have the time in the future, could you please try to run the tests again on this branch with an additional patch that changes: effekt/effekt/jvm/src/test/scala/effekt/ParserTests.scala Lines 34 to 36 in e15eca1 into: def sourceAndPositions: (Source, Seq[Int]) = {
val normalized = content.stripMargin.replace("\r\n", "\n")
val lines = normalized.split("\n", -1).toBuffer
val positions = = scala.collection.mutable.ArrayBuffer[Int]()
...I looked at the ParserTests and the |
|
@jiribenes I fixed the test (also had to normalize the expected value). I'm still confused by the regex above though. |
| val textWithoutSpan = | ||
| raw"""There is some content here. | ||
| |And here.""".stripMargin | ||
| |And here. | ||
| |""".stripMargin.replace("\r\n", "\n") | ||
|
|
||
| assertEquals(span.from, 4) | ||
| assertEquals(span.to, 33) | ||
| assertEquals(source.content, textWithoutSpan) |
There was a problem hiding this comment.
alt. we could use assertNoDiff(source.content, textWithoutSpan)
WDYM? 🤔 |
This change should help with resolving #1183, though I can't confirm it's solved completely while being on a Mac.
I also changed the Windows smoke tests to run the case studies -- hopefully this means a slightly bigger subset of the language being checked on Windows CI :)