Missing USOCKET::FDSET-ALLOC on CL REPL (EQL5, Andoid) #67

silent-entertainment · 2020-10-30T19:27:29Z

I'm trying to use hunchentoot.

Before loading hunchentoot, *FEATURES* includes :ECL, :ECL-BYTECMP, :EQL, :EQL5 and :ANDROID -to narrow the field for you, though among these I'm aware that usocket system configuration seems to involve only :ECL and :ECL-BYTECOMP, from what I've been able to glean from usocket files.

hunchentoot pulls in usocket, but it raises a missing USOCKET::FDSET-ALLOC when I try to (start ...) hunchentoot per the documentation.

(ql:quickload "hunchentoot")

(in-package 'usocket)

(fboundp 'fdset-alloc)

NIL

When I reported the issue to hunchentoot, hunchentoot told me it was an issue for usocket. Another CL user on another forum also recommended that I report the issue to usocket.

Meanwhile, I'm trying to trace the bug to get this working, but I don't have enough recent CL experience or knowledge about all of the internals to be able to help much yet.

edit: forgot to proofread

silent-entertainment · 2020-10-30T21:17:55Z

From what I've been able to discern, #+ecl pulls in file ecl, which depends on sbcl.

ecl deals with (and #+ecl-bytecmp #+windows), so it comes down to sbcl.

sbcl wants #+(and ecl (not ecl-bytecmp)) to define fdset-alloc, so we're out of luck.

usocket.asd defines

#+sample (pushnew :usocket-iolib features)

but when I tried

(pushnew :sample *features*)

(ql:quickload "hunchentoot")

it choked on iolib.

edit: fixed markdown

binghe · 2020-10-31T03:46:39Z

Hi,

I'm afraid that usocket currently does not (fully) support ECL/Android if in ECL you have only the bytecode compiler.

The function fdset-alloc (and siblings) are used to implement WAIT-FOR-INPUT (required by Hunchentoot), and it's only defined when ECL can call the underlying C compiler. On Windows, as usually there's no C compiler available, we directly use ECL's foreign function interface (FFI) to call the related Win32 APIs, and this works only for :ecl-bytecmp (ECL historically doesn't support bytecode compiler, and it's added later.)

In theory, we should be able to use FFI to re-implement WAIT-FOR-INPUT for ECL on Unix-like systems, following a similar path as some other CL backends. But this needs some time (let me know if you want to wait for several weeks and be a tester.)

But I wonder if can rebuild your ECL with native C compiler on Android? (i.e. :ecl-bytecmp does not appear in *features*). If it cannot be so technically, this will mean that EQL/Andriod cannot use usocket, that's really bad and we must try to fix it (in the above way).

Regards,

Chun Tian

binghe · 2020-10-31T03:49:55Z

I don't know if anyone ever succeeded in loading IOlib on Android, as it requires a dedicated dynamic library (libfixposix) to be successfully built and installed first. Besides, the IOlib backend is highly experimental, currently the implementation of WAIT-FOR-INPUT is still buggy (and I don't know how to fix...).

silent-entertainment · 2020-10-31T17:32:48Z

Hi binghe,

Thank you for your attention.

What can I do to help?

I haven't done a lot of low-level programming. I've tried to dig into the internals of ECL, its bytecode compiler, ffi, etc., and I've looked more closely at usocket and sockets in general. I'm not completely up to speed, but the picture is coming into focus.

Trying iolib was a shot in the dark, to see if it might be a usable workaround. I admit that I was completely ignorant about the internals and actual incompatibility of iolib, and I wasn't too surprised that it didn't work. I mentioned it only to apprise you of what had been tried, to contribute the data points.

Though I realize that Common Lisp and CL REPL have been available for Android for a relatively short time, it's very surprising to me that libray compatibility for major components is so far behind, when Android has such a large market share and considering that so many programmers have jumped on the bandwagon. Of course, CL popularity is relatively low compared to newer boilerplate and bubblegum programming languages, but there is quite a base of CL and other Lispy projects.

In fact, I decided to become reaquainted with programming and specifically with Common Lisp very recently, after an extended hiatus from programming much of anything other than a couple of small JavaScript programs, which happened due to a change in lifestyle and circumstances. An old unfinished AI-related project came into focus, and Common Lisp was the only right choice.

Progress of the aforementioned project is going to require a user interface. Though EQL is integrated with the Qt framework, a browser-based interface and a small web server implemented in CL makes more sense to me. This thinking leads me to the possibilty of usinge hunchentoot, cl-who and parenscript, but now I find that hunchentoot happens to be a roadblock, due to its dependency on usocket and its current compatibility status. Other potential servers are poorly documented, have awkward interfaces or have the dependency on usocket or other unsupported libraries (e.g. the ACL-COMPAT dependency of AllegroServe).

Currently, I have limited computer resources, so rebuilding ECL would be difficult.

You mentioned the possibility of using FFI to implement WAIT-FOR-INPUT. Which backend would be a good reference template on how to do that?

If it's good with you, I'd like to try to contribute the implementation for Android.

edit: added AI-related

stassats · 2020-10-31T18:09:02Z

Could it be cross-compiled? EQL itself is somehow built for android.

Hi stassats,

Yes, it could be cross-compiled. I currently don't have the computer resources. Also, changing the configuration might result in bugs that haven't been addressed for native compiling on Android. EQL5 for Android is a port of EQL5 for Android, obviously. Porting an application like Common Lisp is a huge task. The implementation of EQL5 that is used by the CL REPL App uses the byte compiler, and I just don't know what the reasons for that might be. I haven't come across any information on an implementation for Android that uses a native compiler or an interpreter that would allow using the sbcl usocket backend in the way that it was designed. Porting a Common Lisp implementation is something that it is beyond my experience level.

The primary goal here is to make usocket compatible with an implementation that is available, so that we can set up a web server, so that we can get some work done. Using FFI on the backend as binghe suggested should be the shortest path to the goal.

silent-entertainment · 2020-11-02T20:11:40Z

Maybe I was looking in the wrong direction. Last night I was exploring a project that uses SWI-Prolog, so out of curiosity I looked to see if SWI-Prolog had been ported to Android. SWI-Prolog has been ported and will run in a Linux on Android userland.

I knew about the possibility of running Linux apps on Android because I had used a GIMP package, which pulled in a full X-Windows desktop, but I hadn't explored it further.

After learning about running SWI-Prolog, I naturally turned to look to see if SBCL might have been successfully ported to Android, using the same userland scheme. It has been done, and a binary is available.

I'll need to get a new tablet with more storage, because my current tablet is stretched a little thin. However, I'm hoping that this path might solve my issue with getting sockets working with CL on Android.

binghe · 2020-11-04T12:07:49Z

Meanwhile I'm trying to the following solution: whenever (dynamic) FFI is available, usocket prevents from using inline C code. I'd like to know if *FEATURES* indeed contains anything related to FFI on Android. @silent-entertainment (better if you can paste a full list of available features.)

silent-entertainment · 2020-11-18T02:16:14Z

binghe,

i didn't see your message. It does have several feature with FFI.

On startup, *FEATURES* is attached.

QUICKLISP, ASDF-PACKAGE-SYSTEM, ASDF3.1, ASDF3, ASDF2, ASDF, OS-UNIX, NON-BASE-CHARS-EXIST-P, ASDF-UNICODE, ECL-BYTECMP, CL-PPCRE, SPLIT-SEQUENCE, QT-WRAPPER-FUNCTIONS, EQL5, EQL, WALKER, CDR-1, CDR-5, ANDROID, FORMATTER, CDR-7, ECL-WEAK-HASH, LITTLE-ENDIAN, ECL-READ-WRITE-LOCK, LONG-LONG, UINT64-T, UINT32-T, UINT16-T, LONG-FLOAT, UNICODE, DFFI, CLOS-STREAMS, CMU-FORMAT, UNIX, ECL-PDE, DLOPEN, CLOS, THREADS, BOEHM-GC, ANSI-CL, COMMON-LISP, FLOATING-POINT-EXCEPTIONS, IEEE-FLOATING-POINT, PACKAGE-LOCAL-NICKNAMES, CDR-14, PREFIXED-API, FFI, ARM, COMMON, ECL

After loading hunchentoot, *FEATURES* is as follows:
HUNCHENTOOT, FLAT-NAMESPACE, UNIX, CFFI, FLAT-NAMESPACE, 64-BIT, FLEXI-STREAMS, CL-FAD, BORDEAUX-THREADS, CHUNGA, THREAD-SUPPORT, QUICKLISP, ASDF-PACKAGE-SYSTEM, ASDF3.1, ASDF3, ASDF2, ASDF, OS-UNIX, NON-BASE-CHARS-EXIST-P, ASDF-UNICODE, ECL-BYTECMP, CL-PPCRE, SPLIT-SEQUENCE, QT-WRAPPER-FUNCTIONS, EQL5, EQL, WALKER, CDR-1, CDR-5, ANDROID, FORMATTER, CDR-7, ECL-WEAK-HASH, LITTLE-ENDIAN, ECL-READ-WRITE-LOCK, LONG-LONG, UINT64-T, UINT32-T, UINT16-T, LONG-FLOAT, UNICODE, DFFI, CLOS-STREAMS, CMU-FORMAT, UNIX, ECL-PDE, DLOPEN, CLOS, THREADS, BOEHM-GC, ANSI-CL, COMMON-LISP, FLOATING-POINT-EXCEPTIONS, IEEE-FLOATING-POINT, PACKAGE-LOCAL- 9695 NICKNAMES, CDR-14, PREFIXED-API, FFI, ARM, COMMON, ECL

binghe · 2020-11-18T14:40:17Z

Thanks for the valuable information, now I think your original problem can be resolved by rewriting related inline C code in (D)FFI. Please give me some days.

silent-entertainment · 2020-11-18T15:25:00Z

@binghe
Awesome. No hurries. No worries. Take your time.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing USOCKET::FDSET-ALLOC on CL REPL (EQL5, Andoid) #67

Missing USOCKET::FDSET-ALLOC on CL REPL (EQL5, Andoid) #67

Missing USOCKET::FDSET-ALLOC on CL REPL (EQL5, Andoid) #67

Missing USOCKET::FDSET-ALLOC on CL REPL (EQL5, Andoid) #67

Comments