Ko 11 tools for writing language extensions - several issues

TL;DR
I wrote a simple UDL script a couple of years ago under Ko 9 that worked well. All it needs to do is highlight at the beginning of a line any tags in the form [x] where x is a letter. Ko 10 broke it with no explanation - only that it was “incompatible”. Ko 11 says the same thing when I import it, even though I built it with the Ko 11 SDK.

Also, the Ko 11 IDE tools for building the xpi are broken so I can only use the command line methods outlined for Ko 9 => koext.py build to create the xpi.

My questions are as follows: Have the language extension build tools for Ko 11 been tested? If so, what am I doing wrong. If not, when do you expect them to work? I do not want to spend more time on this if it is something that cannot be done yet.

Details:

If I try to build the xpi using Project - Template - Create Komodo Language I get this

which was identified as a bug in September but never fixed? Also the template directories and files are not created.

As I mentioned above I can build an xpi from the command line using koext.py and the python version provided in the Ko 11 sdk folder. However it is flagged as incompatible if I try to load it. This also occured in Ko 10 but not in Ko 9.

Finally here is the simple UDL file that gets the job done if I can get it compiled for Ko 11. Any ideas on what I should do or how long I should wait for the tools to catch up?

# UDL lexer for once

language once

family markup

initial IN_M_DEFAULT

# Null-transition to get into SSL state
state IN_M_DEFAULT:
/./ : redo, => IN_SSL_DEFAULT

# server side language
#family ssl 

# Specify the min and max styles we'll be referring to.
start_style SSL_DEFAULT
end_style SSL_VARIABLE

# Define what happens when in default (initial) state. 
state IN_SSL_DEFAULT:

# ----------- Entering Comments ---------- #
# Hash-style one line comments
'#' : paint(upto, SSL_DEFAULT),  => IN_SSL_COMMENT_TO_EOL

# ----------- Entering Whitespace ---------- #
# Just stay in whatever style we currently are using.
#/[$WS]+/  : #stay

# ----------- Entering Strings ---------- #
# ----------- Entering Numbers ---------- #
# ----------- Entering Identifiers ------ #
# ----------- Entering Operators -------- #

# ----------- Entering Method Tags ------ #
#/^(\[[^\s\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG

/^\s\s(\[[r\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG
/^\s\s(\[[i\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG
/^\s\s(\[[v\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG
/^\s\s(\[[e\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG
/^\s\s(\[[t\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG
/^(\[[s\\]])/: paint(upto, SSL_DEFAULT), => IN_SSL_TAG

# ---------  Exit Method Tabs -------------------- #
state IN_SSL_TAG:
/\]/ : paint(include, SSL_STRING), => IN_SSL_DEFAULT

state IN_SSL_COMMENT_TO_EOL: 
/[\r\n]/ : paint(upto, SSL_COMMENT), => IN_SSL_DEFAULT  # back to default state

fold "[" SSL_OPERATOR +
fold "]" SSL_OPERATOR -

Regarding the version compatibility, you might look at the minVersion and maxVersion entries in the extension’s install.rdf file (specified separately for IDE vs Edit) to make sure they still make sense. If you originally created the extension in/for Komodo v9, it’s possible the install.rdf has version 9.* set as the maximum version for which it is compatible.

Frankly this mechanic is in fact deprecated and not well maintained. We’ll likely end up removing it and re-implementing it similar to the addon bootstrapper.

Note that you can simply run koext.py build from your working directory to manually invoked the build. Have you tried that?

nathanr,

Maybe you should take out the templates on the next release since they do not work.

Yes I tried the koext.py build as explained above but it generates an incompatible xpi that will not load in Ko 10 and 11. I appreciate rstewart’s idea above and will look at that. I generated a new install rdf using the Ko 11 build code but it is possible that it still includes the old version limitation that he suggested.

If what he suggests is true, and you still keep distributing the sdk machinery for building udl files, you should fix it at the build level since if what he says is right, it will always generate xpi’s that will fail. I will update the thread and file a bug report after I check it out.

rstewart was correct that the install.rdf file template has not been updated since Ko 8 so when the max version is updated the xpi is not rejected when manually loaded. However it still does not work. If the xpi is manually loaded it does not show up in the language list. No errors are thrown.

If I try to reinstall it from the GTA bar then the error file logs the following:

[2017-11-09 21:07:53,566] [ERROR] console-logger: TypeError: item.data is undefined (2) in chrome://scope-packages/content/commands.js:12
Traceback (most recent call last):
  File "chrome://scope-packages/content/commands.js", line 12, in 

[2017-11-09 21:08:47,638] [WARNING] views: Inconsistent view count in view_closed event

I don’t understand this message. My conclusion is that these UDL tools have been broken since Ko 8 and I do not think it is worth it to file a bug report because I think there are a number of fundamental problems. Also, I think it is very misleading and wastes a lot of time to include Ko 11 documentation on a build template that does not work at all and apparently, even if the front end did work, the underlying tools have also been broken for a few years.

@rholland I understand your frustration at what you are bumping into. As someone who has developed and maintained a UDL-based language extension for Komodo since before v6, I can say definitively that I did not run into the types of issues you are describing even as recently as the final v10.x releases. I’m on hold with my language extension project at this point as some of the v11 work the Komodo team did to completely rebuild how code-intel works, etc., breaks some capabilities that are integral to my extension, but the capabilities and tools definitely did work well for me through v10.x. Any breakage that may have existed prior to v11 must have been minor enough that I didn’t encounter it; I can’t say definitively that there wasn’t any breakage, only that I didn’t run into it.

I do know that with each subsequent major version release, I’ve had to update the maxVersion attribute and rebuild the extension in order to be able to install the extension.

@rstewart Komodo 11’s codeintel was a huge undertaking and as such the extension part was not implemented as otherwise we would not have been able to finish it. We are working on extension support in version 11.1, which can be facilitated either through LSP (Language Server Protocol) or by creating your own “service mediator” (essentially our own API). Documentation will follow and we will be sharing more info on this with the Community QA Team soon.

@rholland Have you checked if your addon is showing up under GTA > Packages > Manage > List Installed?

Also, anything in the error log after starting Komodo with the addon installed (drag XPI onto Komodo)?

@rstewart - thanks for the help and encouraging words. i will take a look again at some future point

@nathanr - yes it shows up in list installed - that is where I try a reinstall (after a legacy install) and the error above is thrown. The program does not let me drag xpi onto Komodo if you mean by dragging over any one of the panes. After I do the legacy install I do not see any errors.

The error you shared is unrelated. For dragging you need to drag it onto the editor, not onto the panes.

Can you share your full error log?

@nathanr Yup, and no concerns from here. I may not have worded that as well as I perhaps could have in my not-yet-caffeinated early-morning state; I wasn’t complaining and definitely didn’t intend that to come off in a negative way.

I recognize the extent of the effort involved in the whole reworking of that framework, and that it realistically couldn’t have been done without causing some breakage somewhere, particularly in the areas of add-ons. I was just trying to make sure that others like @rholland understood that my decision to pause development of my extension for now was not because the tools, etc., don’t/haven’t worked (I did in fact update and recompile my extension with one of the early builds of v11) but rather was due to the reworked underlying language framework not yet being ready for us to start adding our own bells and whistles to… which is ultimately what I still believe to be one of Komodo’s real strengths. :thumbsup:

That’s fair, thanks @rstewart :slight_smile: