8000 Support for python3.8 · Issue #47 · pyos/dg · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Support for python3.8 #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pbsds opened this issue Jun 13, 2020 · 10 comments
Open

Support for python3.8 #47

pbsds opened this issue Jun 13, 2020 · 10 comments

Comments

@pbsds
Copy link
pbsds commented Jun 13, 2020

This seems like this is a very hard transition.

They have changed the signature of types.CodeType, adding an extra positional argument called posonlyargcount.
Accounting for this is easy, but they have also made drastic changes to the bytecode format, making it difficult to translate the existing bundled bytecode.

My diff so far accounting for the `posonlyargcount` arg in CodeType 
diff --git a/__init__.py b/__init__.py
index 4283df8..099f936 100644
--- a/__init__.py
+++ b/__init__.py
@@ -17,9 +17,14 @@ def load():
             for code in load(fd):
                 eval(code)
     except FileNotFoundError:
+        def _py37to38_CodeType_shim(*args):
+            if len(args) == 15:  # py3.7 and lower
+                args = (args[0], 0, *args[1:])  # insert a posonlyargcount=0
+            return types.CodeType(*args)
+
         try:
             with open(os.path.join(BUNDLE_DIR, PY_TAG + '.dgbundle.py')) as fd:
-                for code in eval(fd.read(), {'C': types.CodeType}):
+                for code in eval(fd.read(), {'C': _py37to38_CodeType_shim}):
                     eval(code)
         except FileNotFoundError:
             raise ImportError('python implementation {!r} not supported'.format(PY_TAG))
diff --git a/core/__init__.dg b/core/__init__.dg
index ea2f5ca..f6edcbc 100644
--- a/core/__init__.dg
+++ b/core/__init__.dg
@@ -55,9 +55,9 @@ save = code tag dirname ->
 
 _repr = x -> if
   x :: types.CodeType =>
-    fields  = ['argcount', 'kwonlyargcount', 'nlocals', 'stacksize', 'flags', 'code', 'consts', 'names']
+    fields  = ['argcount', 'posonlyargcount', 'kwonlyargcount', 'nlocals', 'stacksize', 'flags', 'code', 'consts', 'names']
     fields += ['varnames', 'filename', 'name', 'firstlineno', 'lnotab', 'freevars', 'cellvars']
-    'C({})'.format $ ','.join $ map (f -> _repr $ getattr x $ 'co_' + f) fields
+    'C({})'.format $ ','.join $ map (f -> _repr $ getattr x $ 'co_' + f) $ filter (f -> hasattr x $ 'co_' + f) fields
   x :: list  => '[{}]'.format $ ''.join $ map (i -> _repr i + ',') x
   x :: tuple => '({})'.format $ ''.join $ map (i -> _repr i + ',') x
   otherwise => repr x

I tried my hand at using decompyle3, but in my attempts it failed parsing any dogelang bytecode from python 3.7.

If my understanding of the dg compiler is correct, it compiles directly to bytecode, making the current implementation not work on py3.8+ anyway.
@pyos
Copy link
Owner
pyos commented Jun 13, 2020 •
edited
Loading

Shouldn't actually be all that hard...looks like there are only two major changes:

  • finally handling is now sort of like Java's old jsr opcode - need to insert the new CALL_FINALLY opcode before every outbound jump/return;
  • the compiler is now expected to keep track of stack depth (which is already done) and insert an appropriate amount of pops instead of using BREAK_LOOP and CONTINUE_LOOP's automatic stack unwinding

Not sure what's up with END_ASYNC_FOR, and MAP_ADD is not used (it's for dict comprehensions).

A bigger problem is that dg is way behind on features, e.g. string interpolation and parameter/return annotations.

I tried my hand at using decompyle3, but in my attempts it failed parsing any dogelang bytecode from python 3.7.

There's no point in doing that, yeah. Once the compiler supports 3.8, a new bundle for it can be generated by running on 3.7 with -b cpython-38 030800F0 or something like that.

pyos added a commit that referenced this issue Jun 14, 2020
@pyos
Manually unwind the stack instead of using SETUP_LOOP
7d75981 
SETUP_LOOP got removed in 3.8, and was always redundant anyway.

See #47
pyos added a commit that referenced this issue Jun 14, 2020
@pyos
Rewrite except generation to ease transition to 3.8.
a62fa95 
See #47
pyos added a commit that referenced this issue Jun 14, 2020
@pyos
Rewrite except generation to ease transition to 3.8.
657d92d 
See #47
@pyos
Copy link
Owner
pyos commented Jun 14, 2020

Heh, I looked through the current version of ceval.c and not only CALL_FINALLY is about the same as JVM's jsr, but it'll share jsr's fate as well: https://bugs.python.org/issue33387

@LeaveNhA
Copy link

Anyone who wants to implement the desired changes to fix that issue so the language can continue to live? Version restriction is killing it.

If someone interested, please ping me and we can brainstorm and solve it. Thank you.

@pbsds
Copy link
Author
pbsds commented Jun 25, 2022 •
edited
Loading

I did spend a couple weeks trying to convert dg to be a transpiler in fall 2020, as the python AST representation is not as much of a moving target as the cpython vm is. Since python 3.8 there have been many more major changes to the vm.

https://github.com/pbsds/dg/tree/transpiler

I stopped near close to the end when i realized i had to resort to litter the closures with lots of intermediate variables to ensure correct execution order. This is not transparent to the program, as these hidden variables will show up when calling locals, or when inspecting the resulting code objects.

Since i stopped, the python AST has in preparation of PEP-638 gotten some features that could make the transpilation work better, but i believe a better approach would be to adapt the current tokenizer and parser to work as a frontend to hissp. One such example is hebigo, whose implementation seems to be quite small.

@LeaveNhA
Copy link
LeaveNhA commented Jul 1, 2022 •
edited
Loading

I have two options here to continue;

  • As the original author decided, I can continue to follow his direction and will play the puss-in-the corner with the esoteric CPython VM.
  • Grasp your approach and contribute your way to finish at once.

What would you suggest me, @pbsds?

@pbsds
Copy link
Author
pbsds commented Jul 4, 2022

This is ultimately an exercise you'll most likely be doing, go for the direction you find interesting.
The first direction is a great opportunity to learn about the python VM and about stack machines in general.
The other direction is an adventure into the python AST bindings, or the land of lisp.

@LeaveNhA
Copy link
LeaveNhA commented Jul 5, 2022

Thanks for your comment! I will inform you fellas here when I decide.

@pbsds
Copy link
Author
pbsds commented Aug 16, 2022

Someone on lobste.rs linked to this: https://github.com/rocky/python-xasm, might be viable?

@LeaveNhA
Copy link

Hmm, I really wanna do this. But I cannot proceed further because I need to gather my goals. If I took this language and get back to the design phase, I need to write at least 2 paper over it. Dg over semantic perspective and a short comparative analysis with Dg and Python.

I will be decided in next two weeks. I'm distracted enough until then.

@LeaveNhA
Copy link
LeaveNhA commented Jun 5, 2023

Obviously, I decided not do it.

But I still open to discuss and communicate about the language. Feel free to write to me! 🙋🏻‍♂️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pbsds @pyos @LeaveNhA
0