What is p5.js?

Before we begin, let me briefly introduce p5.js. If you are a front-end developer, or interested in creative programming, you must have heard of it.

p5.js is a JavaScript library that allows us to create various graphics, animations, and interactive effects with very simple and intuitive code. The goal of p5.js is to make programming more accessible and expressive, especially for artists, designers, educators, and beginners.

Why combine large models with p5.js?

You may ask, why should p5.js (https://github.com/processing/p5.js) be associated with large models?

In fact, there are two reasons behind this:

1. 1.  Lower the threshold for creative programming : Although p5.js is very simple, it still has a certain learning cost for people who have no programming foundation at all. If we can use natural language to describe the visual effects we want and let the big model automatically generate p5.js code, then wouldn’t everyone be able to become an artist?
2. 2.  Explore new ways of interaction : Large models can not only understand natural language, but also generate various creative content. If we can combine the “imagination” of large models with the expressiveness of p5.js, we can create unprecedented interactive experiences!

My idea: MCP + WebSocket

Inspired by Unity MCP Package, I came up with a solution to connect large models with p5.js: MCP + WebSocket .

Overall architecture

1. 1.  LLM/Client : This can be any source that can send text commands, such as Claude, Cursor, or a simple command line interface.
2. 2.  Python MCP Server :
   • • Acts as a WebSocket server, receiving commands from LLM/Client.
   • • Parse the instructions and extract key parameters (such as color, shape, position, etc.).
   • • Send parameters to the p5.js Sketch via a WebSocket message.
3. 3.  p5.js Sketch :
   • • Contains a WebSocket client that connects to the Python MCP Server.
   • • Receive messages from the server and update the drawing based on the message content.
   • • Display the drawing results in the browser.
4. 4.  Display : The user’s browser, used to display the output of the p5.js Sketch.

Why WebSocket?

• •  Real-time : WebSocket provides two-way, real-time communication capabilities, which is very suitable for scenarios that require frequent interactions.
• •  Cross-platform : WebSocket is a Web standard and can be used on various platforms and in various languages.
• •  Easy to use : Both Python and JavaScript have mature WebSocket libraries, which makes development very easy.

 Proof of Concept: Code Implementation

To verify the feasibility of this idea, I made a very simple prototype.

Python MCP Server (Simplified Version)

# server.py
import asyncio
import websockets
import json

asyncdefhandler(websocket, path):
    print(f"New connection from {websocket.remote_address}")
    try:
        asyncfor message in websocket:
            print(f"Received: {message}")
            try:
                data = json.loads(message)
                # 假设指令格式: {"command": "draw", "shape": "circle", "x": 100, "y": 100, "color": "red"}
                if data.get("command") == "draw":
                    # 简单起见，直接将收到的数据转发给 p5.js
                    await websocket.send(json.dumps(data))
            except json.JSONDecodeError:
                print("Invalid JSON format")
    except websockets.exceptions.ConnectionClosedError:
        print("Connection closed")

asyncdefstart_server():
    server = await websockets.serve(handler, "localhost", 6500)
    print("WebSocket server started on ws://localhost:6500")
    await server.wait_closed()

if __name__ == "__main__":
    asyncio.run(start_server())

This code implements a simple WebSocket server that listens on port 6500 on the local computer. When receiving a message from a client, it tries to parse the JSON formatted data and, if it contains  "command": "draw", forwards the entire data to the connected p5.js client.

p5.js Sketch (Simple version)

// sketch.js
let socket;

functionsetup() {
createCanvas(400, 400);
  socket = newWebSocket("ws://localhost:6500");

  socket.onopen = function(event) {
    console.log("Connected to server");
  };

  socket.onmessage = function(event) {
    let data = JSON.parse(event.data);
    console.log("Received from server:", data);
    if (data.command === "draw") {
      drawShape(data.shape, data.x, data.y, data.color);
    }
  };

  socket.onclose = function(event) {
    console.log("Disconnected from server");
  };
}

functiondrawShape(shape, x, y, color) {
fill(color);
if (shape === "circle") {
    circle(x, y, 50);
  } elseif (shape === "square") {
    rect(x, y, 50, 50);
  }
}

functiondraw() {
// 可以根据需要添加动画或其他交互
}

This p5.js code creates a WebSocket client and connects to the local port 6500 (which is our Python server). When receiving a message from the server, it will parse the JSON data, and if  command it is correct  draw, it will call  drawShape the function to draw the corresponding graphics.

How it works

1. 1.  Start the Python server : Run in the command line  python server.py.
2. 2.  Open the p5.js sketch : Put  sketch.js the code into an HTML file and then open the HTML file in a browser.
3. 3.  Send instructions : Now, you can use any tool that can send WebSocket messages (even a simple Python script) to  ws://localhost:6500 send JSON formatted instructions to the client, for example:

{"command": "draw", "shape": "circle", "x": 100, "y": 100, "color": "red"}

You should see a red circle on the p5.js canvas!

Future Outlook

This is just a very basic prototype, but it proves that it is possible to connect large models with p5.js using the MCP protocol!

Next, we can:

• •  Improve the instruction set : support more graphics, properties, animation effects, etc.
• •  Integrate LLM : Connect this prototype with LLMs such as Claude to achieve true natural language control.
• •  Error handling : Add more robust error handling mechanism.
• •  Two-way communication : Allow p5.js to send messages to the server as well (e.g., report current status, trigger events, etc.).
• •  User Interface : Develop a more friendly user interface to facilitate user configuration and use.
• •  Packaging : Package the Python server and p5.js code into a library or tool that is easier to use.

If you are interested in this direction, welcome to communicate and explore together! Perhaps, in the near future, we will really be able to use natural language to create all kinds of amazing visual art!