Note: 40

Emacs in SolveIt

(by algal0 on discord, @alexisgallagher on X.)

Note: 496

The SolveIt Linux VM does not give users sudo access to install system software, and it comes with vi pre-installed as its editor.

This dialog shows how to install emacs without sudo.

emacs looks like a text editor, which folks use every day for editing, just like vim, VSCode, etc.. But this appearance is misleading. Emacs is one of the most ancient, most hackable, most protean programming environments in the world. It is powered by a live interactive runtime which lets you download packages, see all source code, and even modify it on the fly. It runs the original language of AI, lisp. It is in fact an all-devouring integration environment, like SolveIt. In many ways it's a spiritual ancestor of SolveIt, so they should get along, much like any decent man should respect his grandfather.

So this dialog will show not only how to install emacs but also some other ways to use emacs in SolveIt. It shows:

How to install emacs, so you can use it as your editor in the SolveIt terminal.
How to run emacs as a daemon, and connect to it from SolveIt, so you can evaluate lisp expressions from Solveit.
How to define AI tools so SolveIt AI can drive emacs, viewing and modifying emacs buffers, so it can help with text edting, and see any other content in emacs -- org-mode files, source code files, live REPLs from various languages, documentation, etc.

Note: 6

Installing emacs

Note: 69

We will rely on this AppImage version of emacs, since it can be installed without any sudo permissions. This is a complete, standard version of emacs. It has merely been, miraculously, well-packaged.

Code: 12 ()

import subprocess
from pathlib import Path

Code: 793 ()

install_emacs_scr="""
#!/usr/bin/env bash
set -euxo pipefail

mkdir -p $HOME/.local/bin

cd $HOME/.local/bin

# github repo: https://github.com/blahgeek/emacs-appimage/releases?q=Release&expanded=true

# download nox release: 

# older: known good
#curl -L -O https://github.com/blahgeek/emacs-appimage/releases/download/github-action-build-11538673229/Emacs-30.0.92-nox.AppImage
#E=Emacs-30.0.92-nox.AppImage

# # newer: untested
# curl -L -O https://github.com/blahgeek/emacs-appimage/releases/download/github-action-build-15043247736/Emacs-30.1-nox-x86_64.AppImage
# E=Emacs-30.1-nox-x86_64.AppImage

curl -L -O https://github.com/blahgeek/emacs-appimage/releases/download/github-action-build-17008212441/Emacs-30.2-nox-x86_64.AppImage
E=Emacs-30.2-nox-x86_64.AppImage

chmod u+x $E
./"$E" --appimage-extract
mkdir emacsdir
mv squashfs-root emacsdir

cat << 'EOF' > ctags
#!/bin/bash
WRAPPER_ROOT="$(dirname "$(readlink -f "$0")")"
exec "$WRAPPER_ROOT/emacsdir/squashfs-root/AppRun" --emacs-appimage-run-as ctags "$@"
EOF

cat << 'EOF' > ebrowse
#!/bin/bash
WRAPPER_ROOT="$(dirname "$(readlink -f "$0")")"
exec "$WRAPPER_ROOT/emacsdir/squashfs-root/AppRun"  --emacs-appimage-run-as ebrowse "$@"
EOF

cat << 'EOF' > emacs
#!/bin/bash
WRAPPER_ROOT="$(dirname "$(readlink -f "$0")")"
exec "$WRAPPER_ROOT/emacsdir/squashfs-root/AppRun" "$@"
EOF

cat << 'EOF' > emacsclient
#!/bin/bash
WRAPPER_ROOT="$(dirname "$(readlink -f "$0")")"
exec "$WRAPPER_ROOT/emacsdir/squashfs-root/AppRun"  --emacs-appimage-run-as emacsclient "$@"
EOF

chmod u+x ctags ebrowse emacs emacsclient
"""

Code: 124 ()

def install_emacs():
    "Installs emacs, returning True on success"
    from pathlib import Path
    if (Path().home() / '.local/bin/emacs').exists():
        print("already installed.")
        return True
    cp = subprocess.run(install_emacs_scr,shell=True,executable='/usr/bin/bash',capture_output=True)
    return cp.returncode == 0

Code: 6 ()

install_emacs()

Output: 36

already installed.

True

Note: 90

Now that emacs is installed, you can open the SolveIt terminal and run emacs from the command line to edit any file. You might need to add ~/.local/bin to your PATH.

However, we can do much more if we run emacs as a server.

Note: 9

Running an emacs server

Note: 153

Running emacs as a server allows you to run the editor once, keep it in memory, and then connect to it whenever you want to edit a file or do other work. This gives you a long-running editing session with instant startup.

This is also known as "daemon mode". We'll use it so that we and Solveit can both connect to the same emacs instance at the same time.

Code: 82 ()

def start_emacs_server():
    "Starts an emacs daemon"
    import subprocess
    from pathlib import Path
    return subprocess.run(str(Path().home() / '.local/bin/emacs') + ' --daemon',shell=True,executable='/usr/bin/bash')

Code: 7 ()

start_emacs_server()

Output: 142

Starting Emacs daemon.
Unable to start the daemon.
Another instance of Emacs is running the server, either as daemon or interactively.
You can use emacsclient to connect to that Emacs process.
Error: server did not start correctly

CompletedProcess(args='/app/data/.local/bin/emacs --daemon', returncode=1)

Note: 67

Once you started the emacs server, you can open a file in emacs by running the command emacsclient. (Again, you may need to make sure ~/.local/bin is on your path.)

Note: 69

Evaluating lisp from SolveIt

Once the emacs daemon is running, we can also communicate with it programmatically via emacsclient.

The simplest interaction is just to evaluate a lisp form for its value.

Code: 94 ()

def eval_lisp_for_value(code:str):
    "Evaluates an emacs lisp expression and return the resulting value"
    from pathlib import Path
    return subprocess.run([
        str(Path().home() / '.local/bin/emacsclient'),
        '-e',code],capture_output=True).stdout.decode()

Code: 28 ()

print( eval_lisp_for_value("(+ 1 2 3)").strip() )

Output: 18

Code: 27 ()

print( eval_lisp_for_value(""" "Hello, World" """).strip() )

Output: 25

"Hello, World"

Note: 259

As you can see, the lisp machine is evaluating function applications like (+ 1 2 3) and also atomic values like the string "Hello, World". The resulting output is always a valid lisp form, which is why the string evaluates to a string, with quotations included.

But, we're working mostly in Python, so we don't want our outputs as lisp code, not even lisp string literals, which we would have to transform into valid Python strings. We probably just want to print those string literals. More generally, we evaluate many forms only for the side-effect of what they print to stdout or to the emacs Messages buffer as errors and warnings.

So for both purposes, let's write a more capable eval function, which can return not only the value of a form, but also anything it prints to stdout or to messages.

Code: 579 ()

def eval_lisp(code:str):
    "Evaluates a lisp expression and return (value,out,message)"
    from pathlib import Path
    import tempfile
    tmpout = tempfile.NamedTemporaryFile(delete=False).name
    tmperr = tempfile.NamedTemporaryFile(delete=False).name
    preamble = """
(require 'cl-lib) 
(defmacro with-captured-elisp-io (stdout-file stderr-file &rest body)
  "Capture standard-output and message output while returning BODY's value."
  (declare (indent 2))
  `(let ((out (generate-new-buffer " *out*"))
         (err (generate-new-buffer " *err*")))
     (unwind-protect
         (prog1
             (cl-letf (((symbol-function 'message)
                        (lambda (fmt &rest args)
                          (with-current-buffer err
                            (insert (apply #'format fmt args) "\n"))
                          ;; Return the displayed message as usual:
                          (apply #'format fmt args))))
               (let ((standard-output out))
                 ,@body))
           (with-current-buffer out
             (write-region (point-min) (point-max) ,stdout-file nil 'silent))
           (with-current-buffer err
             (write-region (point-min) (point-max) ,stderr-file nil 'silent)))
       (kill-buffer out)
       (kill-buffer err))))"""
    wrapped_code =  f"""(with-captured-elisp-io "{tmpout}" "{tmperr}" {code})"""
    form = "(progn " + preamble + wrapped_code + ")"
    cp = subprocess.run([str(Path().home() / '.local/bin/emacsclient'),'-e',form],capture_output=True)
    out = Path(tmpout).read_text()
    err = Path(tmperr).read_text()
    return cp.stdout.decode(), out, err

Note: 51

We can test this by evaluating a lisp form (princ "Hello\nWorld") which returns the string "Hello\nWorld" but prints the string itself.

Code: 90 ()

ret = eval_lisp(""" (princ "Hello\nWorld") """)
print( "output value: ")
print( ret[0] )
print( "\nprinted output: ")
print( ret[1] )
print( "\nerror messages: ")
print( ret[2] )

Output: 49

output value: 
"Hello\nWorld"


printed output: 
Hello
World

error messages:

Note: 3

Nice.

Note: 100

Defining a lisp magic

We can even do what all the cool kids are doing (👋, DaveTA!), and define a magic for lisp evaluation, in case we wanted to do some glorious lisp hacking. Can you feel the power? It's like electricity shooting through your fingers! 👇⚡️

Code: 109 ()

from IPython.core.magic import register_cell_magic
from dialoghelper import find_msg_id
import base64

@register_cell_magic
def elisp(line, cell):
    """Evaluate emacs lisp in an emacs daemon and return a lisp form."""    
    output = eval_lisp(cell)[0].rstrip()
    if output: print(output)

Code: 19 ()

%%elisp
(+ 1 (* 3 4))

Output: 18

Code: 25 ()

%%elisp
"Hello, World. A string is a lisp atom"

Output: 37

"Hello, World. A string is a lisp atom"

Code: 18 ()

%%elisp
:keywords-are-self-evaluating

Output: 27

:keywords-are-self-evaluating

Code: 7 ()

%%elisp
nil

Output: 18

nil

Note: 6

Let's get whacky

Code: 484 ()

%%elisp
(let* ((width 50)
       (height 50)
       (max-iter 80)
       (x-min -2.5)
       (x-max 1.0)
       (y-min -1.5)
       (y-max 1.5)
       (chars " .:-=+*#%@"))
  (with-output-to-temp-buffer "*m*"
    (dotimes (j height)
      (let* ((y (+ y-min
                   (* (/ (float j) (1- height))
                      (- y-max y-min)))))
        (dotimes (i width)
          (let* ((x (+ x-min
                       (* (/ (float i) (1- width))
                          (- x-max x-min))))
                 (zx 0.0)
                 (zy 0.0)
                 (iter 0))
            (while (and (< iter max-iter)
                        (< (+ (* zx zx) (* zy zy)) 4.0))
              (let* ((zx2 (- (* zx zx) (* zy zy)))
                     (zy2 (* 2.0 zx zy)))
                (setq zx (+ zx2 x)
                      zy (+ zy2 y)
                      iter (1+ iter))))
            (let* ((len (length chars))
                   (index (if (= iter max-iter)
                              (1- len)
                            (floor (* (/ (float iter) max-iter)
                                      (1- len))))))
              (princ (aref chars index)))))
        (princ "\n")))))

Output: 18

nil

Note: 27

Hmm. I wonder what we just put into the buffer called *m*...?

Note: 106

Scripting your editor with tools

This is amusing but if we wanted to do heavyweight lisp hacking, we'd probably be using SBCL. Emacs lisp is mainly useful for scripting and customizing emacs.

So now let's use our new bridge to define Python functions which let Solveit see emacs.

Code: 99 ()

def list_buffers():
    "Prints emacs buffers, using list-buffers-noselect"
    return eval_lisp("""(princ (with-current-buffer 
      (list-buffers-noselect) 
      (buffer-substring-no-properties (point-min) (point-max))))
      """)[1]

Code: 9 ()

print( list_buffers() )

Output: 114

    *scratch*               147 Lisp Interaction 
 %* *Messages*              566 Messages         
 %* *Native-compile-Log*     103 LIMPLE           
  * testbuf                  12 Fundamental      
  * testbuf2                 13 Fundamental      
 %  *m*                    5050 Help

Code: 82 ()

def view_buffer(buffer:str):
    "Returns the contents of an emacs buffer"
    return eval_lisp(f"""(princ (with-current-buffer (get-buffer "{buffer}") (buffer-substring-no-properties (point-min) (point-max))))""")[1]

Code: 13 ()

print( view_buffer("*scratch*") )

Output: 72

;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with ‘C-x C-f’ and enter text in its buffer.

Code: 13 ()

print( view_buffer("*m*") )

Output: 2,640

3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232324632323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232324646323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232464358323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323246456443583232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323246646464613232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232324646436464464632323232323232323232323232
3232323232323232323232323232323232323232323232323232325846464364646464455845323264323232323232323232
3232323232323232323232323232323232323232323232323232325864426464646464646464464558323232323232323232
3232323232323232323232323232323232323232323232323232464642646464646464646464646445323232323232323232
3232323232323232323232323232323232323232323232323246436464646464646464646464646446323232323232323232
3232323232323232323232323232323232323232323232323246586464646464646464646464646446463232323232323232
3232323232323232323232323232323232464646464646324643646464646464646464646464646464453232323232323232
3232323232323232323232323232323232324664586458464645646464646464646464646464646464463232323232323232
3232323232323232323232323232323232464664646464614664646464646464646464646464646464583232323232323232
3232323232323232323232323232324646466464646464645864646464646464646464646464646464323232323232323232
3232323232323232323232323232324646466464646464646464646464646464646464646464646445323232323232323232
3232323232323232323232324632464664646464646464646464646464646464646464646464646432323232323232323232
3232323232323232323232324632464664646464646464646464646464646464646464646464646432323232323232323232
3232323232323232323232323232324646466464646464646464646464646464646464646464646445323232323232323232
3232323232323232323232323232324646466464646464645864646464646464646464646464646464323232323232323232
3232323232323232323232323232323232464664646464614664646464646464646464646464646464583232323232323232
3232323232323232323232323232323232324664586458464645646464646464646464646464646464463232323232323232
3232323232323232323232323232323232464646464646324643646464646464646464646464646464453232323232323232
3232323232323232323232323232323232323232323232323246586464646464646464646464646446463232323232323232
3232323232323232323232323232323232323232323232323246436464646464646464646464646446323232323232323232
3232323232323232323232323232323232323232323232323232464642646464646464646464646445323232323232323232
3232323232323232323232323232323232323232323232323232325864426464646464646464464558323232323232323232
3232323232323232323232323232323232323232323232323232325846464364646464455845323264323232323232323232
3232323232323232323232323232323232323232323232323232323232324646436464464632323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323246646464613232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323246456443583232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232464358323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232324646323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232324632323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232
3232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232

Note: 33

Oh, that's what we put in the *m* buffer -- the Mandelbrot set! 🤓

Note: 13

Let's see if it works as a tool.

Prompt: 78

SolveIt, please can you use the &view_buffer tool and the &list_buffers tool to inspect this running image of emacs and tell me if there is a buffer with log outputs from native compilation, and what is in that buffer?

Assistant: 477

Copied!

{
  "id": "toolu_01NPhTdu2pEeyNa5CVLPTi39",
  "call": {
    "function": "list_buffers",
    "arguments": {}
  },
  "result": "'    *scratch*               147 Lisp Interaction \\n %* *Messages*              515 Messages        <TRUNCATED>"
}

Yes, there's a buffer called *Native-compile-Log*. Let me check what's in it:

Copied!

{
  "id": "toolu_01XzHyvXepzSJNbG696bx3d6",
  "call": {
    "function": "view_buffer",
    "arguments": {
      "buffer": "*Native-compile-Log*"
    }
  },
  "result": "'Compiling /app/data/.emacs.d/eln-cache/30.2-d4caaeb0/subr--trampoline-6d657373616765_message_0.eln.<TRUNCATED>"
}

The *Native-compile-Log* buffer contains a single log entry showing that emacs compiled a trampoline function for message to the native compilation cache at /app/data/.emacs.d/eln-cache/30.2-d4caaeb0/subr--trampoline-6d657373616765_message_0.eln.

Note: 4

Great success!

Code: 120 ()

def create_buffer(name:str, contents:str):
    "Create an emacs buffer, populated with contents"
    lisp_escaped = contents.replace('\\', '\\\\').replace('"', '\\"')
    return eval_lisp(f"""
(with-current-buffer (get-buffer-create "{name}")
  (erase-buffer)
  (insert "{lisp_escaped}")
  (current-buffer))""")[0]

Code: 9 ()

print( list_buffers() )

Output: 96

    *scratch*               147 Lisp Interaction 
 %* *Messages*              515 Messages         
 %* *Native-compile-Log*     103 LIMPLE           
  * testbuf                  12 Fundamental      
  * testbuf2                 13 Fundamental

Code: 15 ()

print( view_buffer('testbuf2') )

Output: 22

test"contents

Note: 3

Nice...

Note: 198

TODO / SOMEDAY MAYBE / NEVER:

TOOLS:
- search_buffer tool
- str_replace buffer tool
Add EXAMPLES of practical uses:
- show how to use emacs's jupyter-mode to read and edit the ipynb dialog files directly.
- how to use an ssh reverse tunnel to run the emacs server on SolveIt, and connect to it form local emacsclient. (or tunnel via web sockets?)
- show how to use magit as a rich git UI work in the SolveIt term
- using emacs windows like tmux panes, in SolveIt term

Note: 3

Scratch

Code: 73 ()

print( eval_lisp_capturing_output("""(princ (with-current-buffer 
      (list-buffers-noselect) 
      (buffer-substring-no-properties (point-min) (point-max))))""")[1] )

Output: 69

    *scratch*               147 Lisp Interaction 
 %* *Messages*              515 Messages         
 %* *Native-compile-Log*     103 LIMPLE

Prompt: 78

I know bash exists at /usr/bin/bash and that /usr/bin/sh is a symlink for dash, whcih is underpowered IIRC. Is there an option I a can pass to subprocess.run to cause it to use bash as the shell?

Assistant: 87

Yes! You can use the executable parameter:

Copied!

subprocess.run(install_emacs, shell=True, executable='/usr/bin/bash')

This tells Python to use bash to interpret the command, even though shell=True would normally use /bin/sh.