Package org.opengrok.indexer.analysis.perl

Class PerlSymbolTokenizer

java.lang.Object
org.opengrok.indexer.analysis.JFlexStateStacker
org.opengrok.indexer.analysis.JFlexSymbolMatcher
org.opengrok.indexer.analysis.perl.PerlSymbolTokenizer
All Implemented Interfaces:
JFlexJointLexer, JFlexLexer, JFlexStackingLexer, Resettable, ScanningSymbolMatcher, SymbolMatchedPublisher
public class PerlSymbolTokenizer extends JFlexSymbolMatcher

  • Field Details

  • Constructor Details

    • PerlSymbolTokenizer

      public PerlSymbolTokenizer(Reader in)
      Creates a new scanner
      Parameters:
      in - the java.io.Reader to read input from.

  • Method Details

    • getYYCHAR

      public long getYYCHAR()
      Gets the yychar value.
      Returns:
      yychar

    • getYYEOF

      public int getYYEOF()
      Gets the YYEOF value.
      Specified by:
      getYYEOF in interface JFlexStackingLexer
      Specified by:
      getYYEOF in class JFlexStateStacker
      Returns:
      YYEOF

    • reset

      public void reset()
      Resets the Perl tracked state; Resets the instance to an initial state.
      Specified by:
      reset in interface Resettable

    • offer

      public void offer(String value) throws IOException
      Description copied from interface: JFlexJointLexer
      Passes non-symbolic fragment for processing.
      Parameters:
      value - the excised fragment
      Throws:
      IOException - if an error occurs while accepting

    • offerSymbol

      public boolean offerSymbol(String value, int captureOffset, boolean ignoreKwd) throws IOException
      Description copied from interface: JFlexJointLexer
      Passes a text fragment that is syntactically a symbol for processing.
      Parameters:
      value - the excised symbol
      captureOffset - the offset from yychar where value began
      ignoreKwd - a value indicating whether keywords should be ignored
      Returns:
      true if the value was not in keywords or if the ignoreKwd was true
      Throws:
      IOException - if an error occurs while accepting

    • skipSymbol

      public void skipSymbol()
      Description copied from interface: JFlexJointLexer
      Indicates that something unusual happened where normally a symbol would have been offered.

    • offerKeyword

      public void offerKeyword(String value) throws IOException
      Description copied from interface: JFlexJointLexer
      Passes a text fragment that is syntactically a keyword symbol for processing.
      Parameters:
      value - the excised symbol
      Throws:
      IOException - if an error occurs while accepting

    • startNewLine

      public void startNewLine() throws IOException
      Description copied from interface: JFlexJointLexer
      Indicates that the current line is ended.
      Throws:
      IOException - if an error occurs when handling the EOL

    • disjointSpan

      public void disjointSpan(String className) throws IOException
      Description copied from interface: JFlexJointLexer
      Indicates the closing of an open tag and the opening -- if className is non-null -- of a new one.
      Parameters:
      className - the class name for the new tag or null just to close an open tag.
      Throws:
      IOException - if an output error occurs

    • phLOC

      public void phLOC()
      Description copied from interface: JFlexJointLexer
      Indicates that eligible source code was encountered for physical lines-of-code count (physical LOC).

    • abortQuote

      public void abortQuote() throws IOException
      Throws:
      IOException

    • maybeIntraState

      public void maybeIntraState()

    • takeAllContent

      protected boolean takeAllContent()

    • returnOnSymbol

      protected boolean returnOnSymbol()

    • skipLink

      protected void skipLink(String url, Pattern p)

    • yyclose

      public final void yyclose() throws IOException
      Closes the input reader.
      Throws:
      IOException - if the reader could not be closed.

    • yyreset

      public final void yyreset(Reader reader)
      Resets the scanner to read from a new input stream.

      Does not close the old reader.

      All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.

      Internal scan buffer is resized down to its initial length, if it has grown.

      Parameters:
      reader - The new input stream.

    • yyatEOF

      public final boolean yyatEOF()
      Returns whether the scanner has reached the end of the reader it reads from.
      Returns:
      whether the scanner has reached EOF.

    • yystate

      public final int yystate()
      Returns the current lexical state.
      Returns:
      the current lexical state.

    • yybegin

      public final void yybegin(int newState)
      Enters a new lexical state.
      Parameters:
      newState - the new lexical state

    • yytext

      public final String yytext()
      Returns the text matched by the current regular expression.
      Returns:
      the matched text.

    • yycharat

      public final char yycharat(int position)
      Returns the character at the given position from the matched text.

      It is equivalent to yytext().charAt(pos), but faster.

      Parameters:
      position - the position of the character to fetch. A value from 0 to yylength()-1.
      Returns:
      the character at position.

    • yylength

      public final int yylength()
      How many characters were matched.
      Returns:
      the length of the matched text region.

    • yypushback

      public void yypushback(int number)
      Pushes the specified amount of characters back into the input stream.

      They will be read again by then next call of the scanning method.

      Parameters:
      number - the number of characters to be read again. This number must not be greater than yylength().

    • yylex

      public int yylex() throws IOException
      Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
      Returns:
      the next token.
      Throws:
      IOException - if any I/O-Error occurs.

    • maybeEndQuote

      public boolean maybeEndQuote(String capture)
      Determine if the quote should end based on the first character of capture, recognizing quote-like operators that allow nesting to increase the nesting level if appropriate.

      Calling this method has side effects to possibly modify nqchar, waitq, or endqchar.

      Returns:
      true if the quote state should end

    • areModifiersOK

      public boolean areModifiersOK()
      Gets a value indicating if modifiers are OK at the end of the last quote-like operator.
      Returns:
      true if modifiers are OK

    • qop

      public void qop(String op, int namelength, boolean nointerp) throws IOException
      Starts a quote-like operator as specified in a syntax fragment, op, and gives the op for the lexer to take.
      Throws:
      IOException

    • qop

      public void qop(boolean doWrite, String capture, int namelength, boolean nointerp) throws IOException
      Starts a quote-like operator as specified in a syntax fragment, op, and gives the capture for the lexer to take if doWrite is true.
      Throws:
      IOException

    • setState

      public void setState(String ltpostop, boolean nointerp)
      Sets the jflex state reflecting ltpostop and nointerp.

    • hqopPunc

      public void hqopPunc(String capture) throws IOException
      Begins a quote-like state for a heuristic match of the shorthand // of m// where the capture ends with "/", begins with punctuation, and the intervening whitespace may contain LFs -- and writes the parts to output.
      Throws:
      IOException

    • hqopSymbol

      public void hqopSymbol(String capture) throws IOException
      Begins a quote-like state for a heuristic match of the shorthand // of m// where the capture ends with "/", begins with an initial symbol, and the intervening whitespace may contain LFs -- and writes the parts to output.
      Throws:
      IOException

    • hop

      public void hop(String capture) throws IOException
      Parses a Here-document declaration, and takes the capture using JFlexJointLexer.offer(java.lang.String). If the declaration is valid, hereSettings will have been appended.
      Throws:
      IOException

    • maybeStartHere

      public boolean maybeStartHere() throws IOException
      Pushes the first Here-document state if any declarations were parsed, or else does nothing.
      Returns:
      true if a Here state was pushed
      Throws:
      IOException

    • maybeEndHere

      public boolean maybeEndHere(String capture) throws IOException
      Process the capture, possibly ending the Here-document state just beforehand.
      Returns:
      true if the quote state ended
      Throws:
      IOException

    • sigilID

      public void sigilID(String capture) throws IOException
      Splits a sigil identifier -- where the capture starts with a sigil and ends in an identifier and where Perl allows whitespace after the sigil -- and write the parts to output.

      Seeing the endqchar in the capture will affect an active quote-like operator.

      Throws:
      IOException

    • bracedSigilID

      public void bracedSigilID(String capture) throws IOException
      Splits a braced sigil identifier -- where the capture starts with a sigil and ends with a '}' and where Perl allows whitespace after the sigil and around the identifier -- and write the parts to output.
      Throws:
      IOException

    • specialID

      public void specialID(String capture) throws IOException
      Write a special identifier as a keyword -- unless endqchar is in the capture, which will affect an active quote-like operator instead.
      Throws:
      IOException

    • getCollateralCapturePattern

      public Pattern getCollateralCapturePattern()
      Gets a pattern to match the collateral capture for the current quoting state or null if there is no active quoting state.
      Returns:
      a defined pattern or null

    • chkLOC

      public void chkLOC()
      Calls JFlexJointLexer.phLOC() if the yystate is not SCOMMENT or POD.