I've spent the past week or so chipping away at the problems with my parser. As I started working on the obvious bugs, more holes in the ship revealed themselves, and I've had to think really carefully about what each function inputs and outputs and under what circumstances. I've managed to write a parser that reaches the end of input successfully, and seemingly generates the correct parse tree, but once it reaches the end of input, it doesn't return to the top level function…I think there's some condition that I'm not handling in the passage function that should be handling EOF, but isn't…I believe this would be the condition that allows there to be a passage that isn't followed by any other passage. Once I can get that working, I should be able to move on to the template building part of the transpiler that actually generates the Twee3 source code to be compiled into webpages. Everything else after that point is dealing with different values for tokens rather than generating a tree or processing input directly, so it should be more manageable past this point.
I've taken the approach of printing basically everything. Here's the output so far: once the number of tokens hits zero, the program stalls, so I have to close it using ^Z.
hypatia@Yours-MacBook-Air:https://cs.marlboro.college/cours/fall2019/jims_tutorials/MEGA/plan work/programming_languages$ python3 tweetex.py basictweetex.twx
-------printing tokens produced by lexer
Token('title', 'COMMAND', '\title')
Token('{', 'LEFTCURLY', '{')
Token('My Story', 'CHARACTER', 'My Story')
Token('}', 'RIGHTCURLY', '}')
Token('author', 'COMMAND', '\author')
Token('{', 'LEFTCURLY', '{')
Token('Nick Creel', 'CHARACTER', 'Nick Creel')
Token('}', 'RIGHTCURLY', '}')
Token('ifid', 'COMMAND', '\ifid')
Token('{', 'LEFTCURLY', '{')
Token('0CA8C7C5-F219-4B1B-A3A8-45710F389818', 'CHARACTER', '0CA8C7C5-F219-4B1B-A3A8-45710F389818')
Token('}', 'RIGHTCURLY', '}')
Token('start', 'COMMAND', '\start')
Token('{', 'LEFTCURLY', '{')
Token('Starting Passage', 'CHARACTER', 'Starting Passage')
Token('}', 'RIGHTCURLY', '}')
Token('passage', 'COMMAND', '\passage')
Token('{', 'LEFTCURLY', '{')
Token('Starting Passage', 'CHARACTER', 'Starting Passage')
Token('}', 'RIGHTCURLY', '}')
Token('This is some text in the first passage
', 'CHARACTER', 'This is some text in the first passage
')
Token('link', 'COMMAND', '\link')
Token('{', 'LEFTCURLY', '{')
Token('Second Passage', 'CHARACTER', 'Second Passage')
Token('}', 'RIGHTCURLY', '}')
Token('{', 'LEFTCURLY', '{')
Token('This link goes to the second passage', 'CHARACTER', 'This link goes to the second passage')
Token('}', 'RIGHTCURLY', '}')
Token('passage', 'COMMAND', '\passage')
Token('{', 'LEFTCURLY', '{')
Token('Second Passage', 'CHARACTER', 'Second Passage')
Token('}', 'RIGHTCURLY', '}')
Token('This is some text in the second passage.
', 'CHARACTER', 'This is some text in the second passage.
')
--------beginning parse routine
tokens remaining == 33
in tokenqueue, token is Token('title', 'COMMAND', '\title')
tokens remaining == 32
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 31
in tokenqueue, token is Token('My Story', 'CHARACTER', 'My Story')
appending text to argument.children, value is My Story
tokens remaining == 30
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('My Story', 'CHARACTER', 'My Story')]
tokens remaining == 29
in tokenqueue, token is Token('author', 'COMMAND', '\author')
argument is [Token('None', 'ARGUMENT', 'None')]
returning macro
tokens remaining == 29
in tokenqueue, token is Token('author', 'COMMAND', '\author')
['ifid', 'title', 'start', 'author']
tokens remaining == 28
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 27
in tokenqueue, token is Token('Nick Creel', 'CHARACTER', 'Nick Creel')
appending text to argument.children, value is Nick Creel
tokens remaining == 26
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('Nick Creel', 'CHARACTER', 'Nick Creel')]
tokens remaining == 25
in tokenqueue, token is Token('ifid', 'COMMAND', '\ifid')
argument is [Token('None', 'ARGUMENT', 'None')]
returning macro
tokens remaining == 25
in tokenqueue, token is Token('ifid', 'COMMAND', '\ifid')
['ifid', 'title', 'start', 'author']
tokens remaining == 24
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 23
in tokenqueue, token is Token('0CA8C7C5-F219-4B1B-A3A8-45710F389818', 'CHARACTER', '0CA8C7C5-F219-4B1B-A3A8-45710F389818')
appending text to argument.children, value is 0CA8C7C5-F219-4B1B-A3A8-45710F389818
tokens remaining == 22
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('0CA8C7C5-F219-4B1B-A3A8-45710F389818', 'CHARACTER', '0CA8C7C5-F219-4B1B-A3A8-45710F389818')]
tokens remaining == 21
in tokenqueue, token is Token('start', 'COMMAND', '\start')
argument is [Token('None', 'ARGUMENT', 'None')]
returning macro
tokens remaining == 21
in tokenqueue, token is Token('start', 'COMMAND', '\start')
['ifid', 'title', 'start', 'author']
tokens remaining == 20
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 19
in tokenqueue, token is Token('Starting Passage', 'CHARACTER', 'Starting Passage')
appending text to argument.children, value is Starting Passage
tokens remaining == 18
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('Starting Passage', 'CHARACTER', 'Starting Passage')]
tokens remaining == 17
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
argument is [Token('None', 'ARGUMENT', 'None')]
returning macro
tokens remaining == 17
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
['ifid', 'title', 'start', 'author']
<lexer.Queue object at 0x10da64ed0>
tokens remaining == 17
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
in passage, token value is passage, type is COMMAND
in passage while loop, Token('passage', 'COMMAND', '\passage') COMMAND
tokens remaining == 16
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 15
in tokenqueue, token is Token('Starting Passage', 'CHARACTER', 'Starting Passage')
appending text to argument.children, value is Starting Passage
tokens remaining == 14
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('Starting Passage', 'CHARACTER', 'Starting Passage')]
tokens remaining == 13
in tokenqueue, token is Token('This is some text in the first passage
', 'CHARACTER', 'This is some text in the first passage
')
tokens remaining == 13
in tokenqueue, token is Token('This is some text in the first passage
', 'CHARACTER', 'This is some text in the first passage
')
tokens remaining == 12
in tokenqueue, token is Token('link', 'COMMAND', '\link')
running macro for _macro in text
tokens remaining == 11
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 10
in tokenqueue, token is Token('Second Passage', 'CHARACTER', 'Second Passage')
appending text to argument.children, value is Second Passage
tokens remaining == 9
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('Second Passage', 'CHARACTER', 'Second Passage')]
tokens remaining == 8
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
tokens remaining == 7
in tokenqueue, token is Token('This link goes to the second passage', 'CHARACTER', 'This link goes to the second passage')
appending text to argument.children, value is This link goes to the second passage
tokens remaining == 6
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('Second Passage', 'CHARACTER', 'Second Passage'), Token('This link goes to the second passage', 'CHARACTER', 'This link goes to the second passage')]
tokens remaining == 5
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
argument is [Token('None', 'ARGUMENT', 'None'), Token('None', 'ARGUMENT', 'None')]
returning macro
tokens remaining == 5
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
done with text
tokens remaining == 5
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
if _text, passage
in else statement of grab text
passage is Token('None', 'PASSAGE', 'None')
tokens remaining == 5
in tokenqueue, token is Token('passage', 'COMMAND', '\passage')
in passage, token value is passage, type is COMMAND
in passage while loop, Token('passage', 'COMMAND', '\passage') COMMAND
tokens remaining == 4
in tokenqueue, token is Token('{', 'LEFTCURLY', '{')
in arg after next token, token.token_type is LEFTCURLY
token.value is {
tokens remaining == 3
in tokenqueue, token is Token('Second Passage', 'CHARACTER', 'Second Passage')
appending text to argument.children, value is Second Passage
tokens remaining == 2
in tokenqueue, token is Token('}', 'RIGHTCURLY', '}')
looking for right curly, token type is RIGHTCURLY
Token('None', 'ARGUMENT', 'None') [Token('Second Passage', 'CHARACTER', 'Second Passage')]
tokens remaining == 1
in tokenqueue, token is Token('This is some text in the second passage.
', 'CHARACTER', 'This is some text in the second passage.
')
tokens remaining == 1
in tokenqueue, token is Token('This is some text in the second passage.
', 'CHARACTER', 'This is some text in the second passage.
')
tokens remaining == 0
^Z
At the beginning of the semester, I planned on having a minimum deliverable product by the end of the semester. I haven't met this goal, though I would say I'm around 80% there in terms of progress. In a sense, I'm not super surprised… I started this semester not really knowing what my minimum deliverable product would even look like besides "uh, I want to make interactive literature with it?", and although I took a few weeks just to figure that out, I think it was helpful to see what other people in the same field are doing with their interactive fiction software, and the information I learned during my (sort of) literature review will still be useful for the actual plan paper that will accompany this code.
It seems to me that the most logical thing to do would be to work together with you in tutorial next semester towards finishing this thing and making a write up for the plan paper. I will have some time to work on this over the break, as well as for the rest of the week, so I will be pushing to finish the parsing portion at least before we return; I don't want to be pushing this hard in April, so I need to get to some minimum deliverable product ASAP. If our first meeting next semester is an appropriate deadline in your eyes for that, I would agree. I think it's possible in the next week to finish the parser and get started on the templating portion of the compiler, but I'm not sure how far I could get on the latter until after wrapping up some other projects and assignments for the break. Following the minimum product, I still need to write code to handle all the different macros that a user would want to use when writing a story; I think this will just involve making new rules in the template function to handle different types of commands, which seems simpler than the kind of work I'm doing now, so hopefully it will move faster.
I really do appreciate the feedback you've given me throughout all of this; I know it's been moving along at a snails pace, which has been happening for a variety of reasons…I don't really want to make any excuses, but I've not been working at top capacity in any of my courses this semester. However, I have been trying to maintain some momentum though rather than freezing, which I think(??) I've been successful at, even though it feels like the more I dig, the more dirt gets thrown into the hole.
![[paper clip]](/cours/static/images/paper_clip_tilt.png)
| last modified | size | ||
 |
LICENSE | Sun Dec 22 2024 09:33 am | 1.1K |
|
basictweetex.twx | Sun Dec 22 2024 09:33 am | 300B |
 |
lexer.py | Sun Dec 22 2024 09:33 am | 4.3K |
|
parser.py | Sun Dec 22 2024 09:33 am | 6.6K |
|
tweetex.py | Sun Dec 22 2024 09:33 am | 1.2K |